Oct 30 18:00:32.993518 ip-10-0-133-47 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:00:33.426725 ip-10-0-133-47 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:33.426725 ip-10-0-133-47 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:00:33.426725 ip-10-0-133-47 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:33.426725 ip-10-0-133-47 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.435900 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436130 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436134 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436141 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436146 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436150 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436154 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436157 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436161 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:33.436142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436164 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436167 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436171 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436186 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436190 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436196 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436202 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436207 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436211 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436215 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436218 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436221 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436224 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436227 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436230 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436234 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436236 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436240 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436243 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:33.436488 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436245 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436248 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436250 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436253 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436256 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436258 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436262 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436265 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436268 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436271 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436273 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436278 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436281 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436285 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436288 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436292 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436296 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436299 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436303 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436306 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:33.437081 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436309 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436312 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436314 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436317 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436320 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436323 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436325 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436328 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436332 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436334 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436337 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436342 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436347 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436350 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436353 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436356 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436359 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436370 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436373 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:33.437623 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436375 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436379 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436382 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436387 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436391 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436395 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436399 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436402 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436407 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436410 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436413 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436417 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436421 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436424 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436428 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436431 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436433 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436437 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436440 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:33.438205 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436442 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436445 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436448 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436451 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436455 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436458 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436461 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436464 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436467 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436470 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436473 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436476 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436479 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436482 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436487 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436490 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436494 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436497 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436502 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.436509 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:33.438726 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437791 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437810 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437817 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437822 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437827 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437830 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437833 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437836 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437839 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437842 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437845 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437848 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437851 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437854 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437857 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437860 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437864 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437868 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437870 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:33.439269 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437875 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437879 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437883 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437886 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437897 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437900 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437903 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437907 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437910 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437913 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437916 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437920 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437923 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437928 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437961 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437964 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437968 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437971 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437975 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:33.440099 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437977 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437981 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437983 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437987 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437990 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437993 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437996 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.437999 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438002 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438006 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438008 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438011 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438014 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438017 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438019 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438028 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438032 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438037 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438040 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438043 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:33.440654 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438046 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438049 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438052 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438054 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438057 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438061 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438066 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438069 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438073 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438077 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438080 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438083 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438086 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438089 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438092 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438094 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438097 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438100 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438103 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:33.441213 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438106 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438109 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438113 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438116 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438120 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438123 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438126 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438129 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438132 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438135 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438142 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438145 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438149 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438152 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438155 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438157 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438161 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438163 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438167 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438170 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:33.441745 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438175 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438178 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438181 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438185 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438187 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438191 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438194 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438196 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.438200 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439013 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439026 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439033 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439038 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439041 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439045 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439049 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439052 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439055 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439060 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:33.442301 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439065 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439068 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439072 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439077 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439080 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439083 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439086 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439089 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439092 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439095 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439098 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439100 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439113 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439117 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439120 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439123 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439127 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439130 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439133 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439135 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:33.442814 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439138 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439141 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439144 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439147 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439150 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439152 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439155 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439158 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439160 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439163 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439165 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439168 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439172 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439174 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439177 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439181 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439184 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439187 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439190 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439193 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:33.443428 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439196 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439200 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439202 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439206 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439212 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439215 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439219 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439223 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439226 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439229 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439232 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439237 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439239 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439242 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439244 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439247 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439250 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439253 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439255 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:33.444020 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439258 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439261 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439263 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439266 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439269 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439273 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439276 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439280 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439283 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439286 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439289 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439292 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439295 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439298 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439302 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439305 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439307 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439313 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439316 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439318 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:33.444568 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439321 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439326 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439329 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439332 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439335 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439339 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439341 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439346 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439348 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439351 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439354 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439357 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439359 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439362 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439365 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439368 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.439371 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441306 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441349 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441366 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:00:33.445245 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441374 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441382 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441387 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441396 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441403 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441407 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441411 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441417 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441422 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441429 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441433 2576 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441437 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441441 2576 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441445 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441450 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441456 2576 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441459 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441464 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441469 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441476 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441484 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441488 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441491 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:00:33.445764 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441495 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441500 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441510 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441514 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441518 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441522 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441528 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441533 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441536 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441540 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441545 2576 flags.go:64] FLAG: --enable-server="true" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441549 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441559 2576 flags.go:64] FLAG: --event-burst="100" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441563 2576 flags.go:64] FLAG: --event-qps="50" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441567 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441571 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441575 2576 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441583 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441588 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441593 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441597 2576 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441600 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441605 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441608 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441613 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:00:33.446410 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441617 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441620 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441624 2576 flags.go:64] FLAG: --feature-gates="" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441630 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441634 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441638 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441642 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441646 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441651 2576 flags.go:64] FLAG: --help="false" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441661 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441666 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441670 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441673 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441677 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441682 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441686 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441690 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441693 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441698 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441702 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441705 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441709 2576 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441712 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441715 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:00:33.447205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441718 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441723 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441727 2576 flags.go:64] FLAG: --lock-file="" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441730 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441735 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441739 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441749 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441753 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441758 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441762 2576 flags.go:64] FLAG: --logging-format="text" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441767 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441773 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441777 2576 flags.go:64] FLAG: --manifest-url="" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441781 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441788 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441792 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441797 2576 flags.go:64] FLAG: --max-pods="110" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441803 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441807 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441810 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441814 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441818 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441821 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441825 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441839 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:00:33.447888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441844 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441848 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441852 2576 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441856 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441859 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441863 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441867 2576 flags.go:64] FLAG: --port="10250" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441871 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441875 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f35d4c477ea95aee" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441880 2576 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441885 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441888 2576 flags.go:64] FLAG: --register-node="true" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441892 2576 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441897 2576 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441901 2576 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441905 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441910 2576 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441915 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441920 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441925 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441928 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441958 2576 flags.go:64] FLAG: --runonce="false" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441961 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441965 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441974 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441978 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441982 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:00:33.448675 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441986 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441989 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441993 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.441997 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442000 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442004 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442007 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442010 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442014 2576 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442017 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442024 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442027 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442035 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442040 2576 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442043 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442047 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442051 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442055 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442058 2576 flags.go:64] FLAG: --v="2" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442077 2576 flags.go:64] FLAG: --version="false" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442084 2576 flags.go:64] FLAG: --vmodule="" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442090 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442104 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442304 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:33.449449 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442310 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442314 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442317 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442321 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442325 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442334 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442337 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442341 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442344 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442348 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442350 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442353 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442356 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442359 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442362 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442365 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442368 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442372 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442375 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:33.450141 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442378 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442386 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442389 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442392 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442395 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442398 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442402 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442405 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442408 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442413 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442416 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442419 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442423 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442426 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442429 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442432 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442434 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442439 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442445 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:33.450715 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442449 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442452 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442455 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442457 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442461 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442464 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442467 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442471 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442475 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442479 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442482 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442486 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442489 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442492 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442499 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442502 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442505 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442508 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442511 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:33.451368 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442513 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442517 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442521 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442524 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442527 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442530 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442533 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442537 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442540 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442543 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442545 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442548 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442559 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442565 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442569 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442573 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442576 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442580 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442583 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442588 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:33.452189 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442591 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442594 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442597 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442600 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442604 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442607 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442611 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442619 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442623 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442626 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442628 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442631 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442634 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442637 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442641 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442644 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442647 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442650 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442653 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442656 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:33.452778 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442660 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442663 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442666 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442669 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442678 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442681 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442685 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442689 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.442742 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442800 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442809 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442813 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442817 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442821 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442824 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442830 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:33.453658 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442833 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442837 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442840 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442851 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442856 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442859 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442863 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442867 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442871 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442874 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442880 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442883 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442886 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442890 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442893 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442896 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442899 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442903 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442907 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:33.454418 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442910 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442913 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442918 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442921 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442924 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442926 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442961 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442965 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442968 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442971 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442974 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442976 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442979 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442981 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442984 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442987 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442992 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442994 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.442998 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443000 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:33.455239 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443003 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443006 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443012 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443016 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443018 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443021 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443023 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443027 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443029 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443033 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443036 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443040 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443043 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443046 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443051 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443055 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443058 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443062 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443065 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:33.456087 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443067 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443070 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443074 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443077 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443080 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443083 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443086 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443089 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443093 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443102 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443105 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443108 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443112 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443115 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443118 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443121 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443124 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443127 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443130 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:33.456840 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443133 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443135 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443139 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443142 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443145 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443148 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443152 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443155 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443158 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443161 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443164 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443167 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443170 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443173 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443176 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443179 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443182 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443185 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443188 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:33.457404 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443191 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:33.458116 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443194 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:33.458116 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.443197 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:33.458116 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.458084 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:00:33.458230 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.458132 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:00:33.458230 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.458167 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:00:33.458230 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.458174 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458276 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458286 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458291 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458299 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458303 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458307 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458310 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458313 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458317 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458321 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458326 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458329 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458335 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458341 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458344 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458348 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458352 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458355 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:33.458356 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458359 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458365 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458370 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458374 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458378 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458382 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458384 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458387 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458390 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458393 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458396 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458399 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458402 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458405 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458408 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458411 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458414 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458418 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458420 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:33.458866 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458423 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458426 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458429 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458432 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458435 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458438 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458440 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458443 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458446 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458452 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458456 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458460 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458462 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458468 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458471 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458475 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458479 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458482 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458485 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:33.459586 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458489 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458495 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458499 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458504 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458508 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458511 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458515 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458519 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458522 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458526 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458528 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458531 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458535 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458538 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458541 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458545 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458548 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458551 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458554 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:33.460136 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458557 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458560 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458563 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458565 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458570 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458573 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458576 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458580 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458585 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458588 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458591 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458594 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458599 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458602 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458605 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458607 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458611 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458614 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458617 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458622 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:33.460650 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458625 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458629 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458632 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458636 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458639 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458642 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458647 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458652 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458655 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458659 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.458662 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.458756 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459000 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459008 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459011 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459016 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:33.461254 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459020 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459024 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459027 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459030 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459034 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459038 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459042 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459045 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459048 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459052 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459055 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459060 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459065 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459069 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459072 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459075 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459078 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459081 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459084 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:33.461694 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459087 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459091 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459093 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459097 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459100 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459103 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459105 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459109 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459112 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459114 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459117 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459120 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459124 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459127 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459131 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459136 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459140 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459143 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459149 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:33.462268 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459152 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459155 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459159 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459162 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459165 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459168 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459171 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459174 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459177 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459180 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459183 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459186 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459190 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459193 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459196 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459200 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459203 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459206 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459209 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:33.462775 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459213 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459216 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459219 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459222 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459225 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459228 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459231 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459235 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459238 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459241 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459246 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459250 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459254 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459258 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459261 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459264 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459267 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459270 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459273 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459277 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:33.463442 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459279 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459283 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459287 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459289 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459292 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459295 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459301 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459304 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459308 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459311 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459314 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459316 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459319 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459323 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459326 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459328 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459332 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459335 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459339 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459342 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:33.464201 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459346 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459349 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459353 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459358 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459362 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.459396 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459439 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459445 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459449 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459454 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459458 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459462 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459466 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459469 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459473 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459476 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:33.464740 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459480 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459483 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459486 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459489 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459493 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459499 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459502 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459505 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459508 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459511 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459515 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459519 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459523 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459527 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459530 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459533 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459538 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459543 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459546 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:33.465175 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459550 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459557 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459561 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459564 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459567 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459571 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459574 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459578 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459581 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459584 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459587 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459591 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459594 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459598 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459601 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459605 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459608 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459611 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459614 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459618 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:33.465688 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459621 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459624 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459627 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459630 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459633 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459636 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459648 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459651 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459654 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459657 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459662 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459665 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459669 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459673 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459678 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459682 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459685 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459688 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459691 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:33.466271 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459694 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459697 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459700 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459703 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459706 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459710 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459714 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459717 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459720 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459723 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459726 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459731 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459735 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459739 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459741 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459755 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459758 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459761 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459764 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459772 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:33.466767 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459775 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459778 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459782 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459785 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459788 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459792 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459797 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459801 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459810 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459813 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459817 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459820 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459824 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459828 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459831 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459836 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459840 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:33.459845 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.460004 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.463462 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 18:00:33.467376 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.464925 2576 server.go:1014] "Starting client certificate rotation" Oct 30 18:00:33.467971 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.465106 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:00:33.467971 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.465181 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:00:33.468041 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.467991 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:33.470412 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.470356 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:33.482838 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.482796 2576 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:00:33.488349 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.488302 2576 log.go:25] "Validated CRI v1 image API" Oct 30 18:00:33.490119 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.490062 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:00:33.495047 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.494990 2576 fs.go:135] Filesystem UUIDs: map[16923d44-8629-470f-839d-6b4f569293ae:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f066eb9b-c585-46d3-ae10-ce61e8c725d9:/dev/nvme0n1p3] Oct 30 18:00:33.495234 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.495045 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 18:00:33.503018 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.502745 2576 manager.go:217] Machine: {Timestamp:2026-10-30 18:00:33.500840559 +0000 UTC m=+0.383579105 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096879 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9ef5af269a4be26c0f656ad3a0c6 SystemUUID:ec2a9ef5-af26-9a4b-e26c-0f656ad3a0c6 BootID:a8c99ca4-873b-4c23-9c9f-82e1f1d2eed2 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e8:25:51:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e8:25:51:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:91:fa:f4:7a:73 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:00:33.503018 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.503010 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:00:33.503275 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.503253 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:00:33.507544 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.507448 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:00:33.507803 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.507547 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-47.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:00:33.507856 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.507807 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:00:33.507856 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.507820 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:00:33.507913 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.507879 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:00:33.508804 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.508773 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:00:33.510381 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.510348 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:00:33.515657 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.515625 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:00:33.516338 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.516312 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:00:33.518181 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.518150 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:00:33.518253 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.518210 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:00:33.518253 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.518246 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:00:33.518310 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.518270 2576 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:00:33.518310 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.518292 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:00:33.519160 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.519117 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:00:33.523453 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.523403 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:00:33.531661 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.531603 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:00:33.532874 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.532842 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:00:33.532972 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.532892 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:00:33.533266 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533251 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:00:33.533338 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533274 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:00:33.533338 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533282 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:00:33.533338 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533289 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:00:33.533338 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533296 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:00:33.533338 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533304 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:00:33.533973 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533910 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:00:33.533973 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533950 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:00:33.533973 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533964 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:00:33.533973 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533975 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:00:33.534122 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.533989 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:00:33.534122 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.534001 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:00:33.534864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.534840 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:00:33.534919 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.534869 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:00:33.542263 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.542215 2576 server.go:1263] "Started kubelet" Oct 30 18:00:33.542510 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.542356 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:00:33.542613 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.542484 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:00:33.542662 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.542649 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:00:33.544318 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.544263 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:00:33.544585 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.544544 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:00:33.544756 ip-10-0-133-47 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:00:33.551486 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.551414 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:00:33.553575 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.553501 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 18:00:33.554699 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.554659 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r999l" Oct 30 18:00:33.555010 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.554982 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:00:33.556665 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.556590 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:00:33.557648 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.557613 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:00:33.557784 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.557655 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:00:33.557881 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.557867 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:00:33.558011 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.558000 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:00:33.558011 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.558011 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:00:33.558421 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.558387 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:33.558824 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.558795 2576 factory.go:55] Registering systemd factory Oct 30 18:00:33.558903 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.558830 2576 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:00:33.559269 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.559245 2576 factory.go:153] Registering CRI-O factory Oct 30 18:00:33.559269 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.559268 2576 factory.go:223] Registration of the crio container factory successfully Oct 30 18:00:33.559458 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.559368 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:00:33.559458 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.559418 2576 factory.go:103] Registering Raw factory Oct 30 18:00:33.559458 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.559439 2576 manager.go:1199] Started watching for new ooms in manager Oct 30 18:00:33.560621 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.560582 2576 manager.go:319] Starting recovery of all containers Oct 30 18:00:33.565208 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.565146 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r999l" Oct 30 18:00:33.565836 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.564455 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-47.ec2.internal.18e3608c956bc304 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-47.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-47.ec2.internal,},FirstTimestamp:2026-10-30 18:00:33.542144772 +0000 UTC m=+0.424883317,LastTimestamp:2026-10-30 18:00:33.542144772 +0000 UTC m=+0.424883317,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-47.ec2.internal,}" Oct 30 18:00:33.582484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.582060 2576 manager.go:324] Recovery completed Oct 30 18:00:33.585040 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.584979 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-47.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 18:00:33.585040 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.585010 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 18:00:33.585447 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.585107 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 18:00:33.590296 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.590253 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:33.593380 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.593347 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:33.593502 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.593406 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:33.593502 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.593424 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:33.595002 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.594894 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:33.595002 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.594898 2576 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:00:33.595381 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.595011 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:00:33.595381 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.595058 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:00:33.596418 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.596379 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:33.596635 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.596439 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:33.596635 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.596457 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:33.597740 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.597697 2576 policy_none.go:50] "Start" Oct 30 18:00:33.597907 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.597759 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:00:33.597907 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.597802 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:00:33.600660 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.600628 2576 policy_none.go:44] "Start" Oct 30 18:00:33.629957 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.629861 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:33.633032 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.632997 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:33.665772 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.659344 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:33.665772 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.660234 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:00:33.668167 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.668133 2576 manager.go:341] "Starting Device Plugin manager" Oct 30 18:00:33.668236 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.668219 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:00:33.668281 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.668242 2576 server.go:87] "Starting device plugin registration server" Oct 30 18:00:33.669002 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.668982 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:00:33.669069 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.669009 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:00:33.669181 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.669155 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:00:33.669323 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.669307 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:00:33.669323 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.669321 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:00:33.669664 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.669560 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:00:33.670641 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.670603 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:00:33.670829 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.670710 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:33.672482 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.672431 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:00:33.672482 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.672486 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:00:33.672756 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.672524 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:00:33.672756 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.672606 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 18:00:33.686517 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.686455 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:33.770069 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.770002 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:33.771694 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.771652 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:33.772008 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.771720 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:33.772008 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.771738 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:33.772008 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.771801 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.774202 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.772723 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal"] Oct 30 18:00:33.774345 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.774324 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:33.775988 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.775923 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:33.776263 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.776002 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:33.776263 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.776013 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:33.779059 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.779019 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:33.779239 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.779210 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.779306 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.779290 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:33.780519 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.780481 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:33.780747 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.780537 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:33.780747 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.780549 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:33.780747 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.780481 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:33.780747 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.780614 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:33.780747 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.780626 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:33.783377 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.783321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.783377 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.783391 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:33.785091 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.785050 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:33.785363 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.785114 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:33.785363 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.785132 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:33.791308 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.791246 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.791574 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.791553 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-47.ec2.internal\": node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:33.816358 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.816306 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-47.ec2.internal\" not found" node="ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.823486 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.823416 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-47.ec2.internal\" not found" node="ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.833052 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.833012 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:33.859394 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.859338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.859600 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.859399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05806b328c870eba3d65f2b11ca3bf01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-47.ec2.internal\" (UID: \"05806b328c870eba3d65f2b11ca3bf01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.859600 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.859466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.934342 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:33.934178 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:33.960040 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.959975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.960040 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.960044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.960308 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.960081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05806b328c870eba3d65f2b11ca3bf01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-47.ec2.internal\" (UID: \"05806b328c870eba3d65f2b11ca3bf01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.960308 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.960136 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.960308 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.960145 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:33.960308 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:33.960136 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05806b328c870eba3d65f2b11ca3bf01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-47.ec2.internal\" (UID: \"05806b328c870eba3d65f2b11ca3bf01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:00:34.034497 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.034406 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.122569 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.122495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:34.128349 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.128314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:00:34.135000 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.134935 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.235869 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.235624 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.336224 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.336132 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.442300 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.442196 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.464663 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.464611 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 18:00:34.464984 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.464922 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:00:34.465046 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.464928 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:00:34.465046 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.464929 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:00:34.544191 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.544022 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.575974 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.575807 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:55:33 +0000 UTC" deadline="2028-07-25 07:32:44.829568458 +0000 UTC" Oct 30 18:00:34.575974 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.575933 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15205h32m10.253641422s" Oct 30 18:00:34.644236 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.644168 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.744490 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.744410 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.845597 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:34.845522 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:00:34.883806 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.883744 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:34.959545 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.959471 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:00:34.988012 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.987942 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:34.988394 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:34.988363 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:00:35.012557 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.012511 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:35.028051 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:35.027935 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05806b328c870eba3d65f2b11ca3bf01.slice/crio-81e8fd7cdf67d35f08627b2453021726776de731591739cf1002aa10902d0037 WatchSource:0}: Error finding container 81e8fd7cdf67d35f08627b2453021726776de731591739cf1002aa10902d0037: Status 404 returned error can't find the container with id 81e8fd7cdf67d35f08627b2453021726776de731591739cf1002aa10902d0037 Oct 30 18:00:35.029218 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:35.029163 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfefcd36786ca02a8752b4bece34acc.slice/crio-9359de080b2ab89cf29d6cec50873807f321c40c3a94475298aa812dd4120f93 WatchSource:0}: Error finding container 9359de080b2ab89cf29d6cec50873807f321c40c3a94475298aa812dd4120f93: Status 404 returned error can't find the container with id 9359de080b2ab89cf29d6cec50873807f321c40c3a94475298aa812dd4120f93 Oct 30 18:00:35.037633 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.037589 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:00:35.520652 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.520590 2576 apiserver.go:52] "Watching apiserver" Oct 30 18:00:35.524881 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.524816 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:35.556919 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.556793 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 18:00:35.583763 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.583697 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:35.584346 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.584247 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:00:35.585131 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.585082 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-6w92c","kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal","openshift-cluster-node-tuning-operator/tuned-k8mxn","openshift-image-registry/node-ca-l6vjs","openshift-multus/multus-additional-cni-plugins-gdxmg","openshift-network-operator/iptables-alerter-fl2ts","kube-system/global-pull-secret-syncer-qg2lp","openshift-dns/node-resolver-n97j8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal","openshift-multus/multus-sq5cb","openshift-multus/network-metrics-daemon-hchmm","openshift-network-diagnostics/network-check-target-xpbwl","openshift-ovn-kubernetes/ovnkube-node-g74ls"] Oct 30 18:00:35.588748 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.588682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.589161 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.588889 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:35.592316 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.591874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:35.592316 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.592079 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:35.595193 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.595104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.599892 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.599052 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.612223 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.611874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.614240 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.614145 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" Oct 30 18:00:35.615855 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.614737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.617654 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.617605 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.622794 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.622710 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:35.626471 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.625948 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:35.626471 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.626186 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:35.626471 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.626290 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:00:35.626471 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.626308 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:00:35.626471 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.626311 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:00:35.626471 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.626414 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:00:35.626471 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.626425 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7mh7f" Oct 30 18:00:35.629367 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.629293 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.629671 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.629536 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:00:35.629671 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.629561 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7mh7f" Oct 30 18:00:35.633313 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.633263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.637661 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.637585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.644975 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.644904 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:35.645306 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.645162 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:35.660333 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.660262 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:00:35.670460 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.670399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.670951 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.670902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-socket-dir-parent\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.671116 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.670999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-kubernetes\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.671116 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-var-lib-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.671116 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.671116 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-node-log\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.671349 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-script-lib\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.671349 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-system-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.671349 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-cnibin\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.671349 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:35.671349 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671288 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-netd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.671349 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-config\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-env-overrides\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-host\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d05019b-efc6-444a-986d-6f63420ba3ff-konnectivity-ca\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-tuned\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7906b48-2326-4074-a2a4-a62654c2b785-iptables-alerter-script\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-sys-fs\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-bin\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cni-binary-copy\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-conf-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-serviceca\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671756 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.671829 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krrq\" (UniqueName: \"kubernetes.io/projected/3b8bc620-2f0c-4060-b76e-2b2365e3621f-kube-api-access-6krrq\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671834 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vph5m\" (UniqueName: \"kubernetes.io/projected/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kube-api-access-vph5m\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-etc-kubernetes\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-system-cni-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.671976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-dbus\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-modprobe-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cnibin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-bin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.672496 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7906b48-2326-4074-a2a4-a62654c2b785-host-slash\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx7z4\" (UniqueName: \"kubernetes.io/projected/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-kube-api-access-dx7z4\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysconfig\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-systemd\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-tmp\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-socket-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-daemon-config\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.672978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chz5j\" (UniqueName: \"kubernetes.io/projected/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-kube-api-access-chz5j\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-registration-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-k8s-cni-cncf-io\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.673187 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-netns\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-hostroot\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-host\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7kw\" (UniqueName: \"kubernetes.io/projected/5f811312-1c2a-4099-9cbe-efe1c5063b66-kube-api-access-qw7kw\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-systemd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82127ba1-4bef-43cd-8550-719a4402fcab-ovn-node-metrics-cert\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d05019b-efc6-444a-986d-6f63420ba3ff-agent-certs\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-conf\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn4qt\" (UniqueName: \"kubernetes.io/projected/d7906b48-2326-4074-a2a4-a62654c2b785-kube-api-access-gn4qt\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-netns\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-ovn\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqpm\" (UniqueName: \"kubernetes.io/projected/d4acf9b8-e56f-4958-9742-7303e972ae20-kube-api-access-qqqpm\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-kubelet-config\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-etc-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-kubelet\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.673864 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc620-2f0c-4060-b76e-2b2365e3621f-tmp-dir\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-kubelet\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-slash\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-log-socket\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-run\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-systemd-units\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.673870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-sys\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.674519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-etc-selinux\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.674664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gd4\" (UniqueName: \"kubernetes.io/projected/82127ba1-4bef-43cd-8550-719a4402fcab-kube-api-access-54gd4\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.674783 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.674709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.675305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.674794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-binary-copy\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.675305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.674833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8bc620-2f0c-4060-b76e-2b2365e3621f-hosts-file\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.675305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.674883 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-lib-modules\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.675305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.674921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-device-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.675305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.674993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.675305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.675105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-multus\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.675305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.675271 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-multus-certs\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.675636 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.675330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrg4j\" (UniqueName: \"kubernetes.io/projected/0bbee04d-3f06-4ce5-8020-4d52b9715887-kube-api-access-vrg4j\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:35.675636 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.675380 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.675740 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.675663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-var-lib-kubelet\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.686151 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.685886 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" event={"ID":"05806b328c870eba3d65f2b11ca3bf01","Type":"ContainerStarted","Data":"81e8fd7cdf67d35f08627b2453021726776de731591739cf1002aa10902d0037"} Oct 30 18:00:35.689638 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.689558 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" event={"ID":"9bfefcd36786ca02a8752b4bece34acc","Type":"ContainerStarted","Data":"9359de080b2ab89cf29d6cec50873807f321c40c3a94475298aa812dd4120f93"} Oct 30 18:00:35.723328 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.723268 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" Oct 30 18:00:35.723718 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.723680 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:00:35.723844 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.723772 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" Oct 30 18:00:35.723908 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.723871 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" Oct 30 18:00:35.724034 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.723963 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" Oct 30 18:00:35.724107 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.723972 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:00:35.724172 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.724119 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" Oct 30 18:00:35.724238 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.724218 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" Oct 30 18:00:35.724298 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.724283 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:00:35.724353 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.724304 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" Oct 30 18:00:35.724485 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.724458 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" Oct 30 18:00:35.737569 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.737315 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:00:35.737897 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.737693 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:00:35.737897 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.737844 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:00:35.738101 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.738052 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:00:35.738207 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.738187 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:00:35.738327 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.738309 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:00:35.739343 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.739305 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:00:35.739563 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.739545 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:00:35.739681 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.739661 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:00:35.739826 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.739796 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:00:35.740032 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.739976 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:00:35.741086 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.741022 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:35.743189 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.743143 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:00:35.743743 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.743706 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:00:35.744023 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.743777 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:00:35.744270 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.744236 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:00:35.744408 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.744303 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:35.744408 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.744240 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:00:35.759545 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.759497 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:35.776401 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.776229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-registration-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.776835 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.776806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-k8s-cni-cncf-io\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.777045 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.776468 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-registration-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.777124 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.776903 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-k8s-cni-cncf-io\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.777124 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.776975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-netns\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.777265 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-hostroot\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.777265 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-host\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.777265 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7kw\" (UniqueName: \"kubernetes.io/projected/5f811312-1c2a-4099-9cbe-efe1c5063b66-kube-api-access-qw7kw\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.777364 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-systemd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777364 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777308 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-hostroot\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.777364 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82127ba1-4bef-43cd-8550-719a4402fcab-ovn-node-metrics-cert\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777463 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.777463 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d05019b-efc6-444a-986d-6f63420ba3ff-agent-certs\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:35.777529 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-conf\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.777569 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4qt\" (UniqueName: \"kubernetes.io/projected/d7906b48-2326-4074-a2a4-a62654c2b785-kube-api-access-gn4qt\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.777569 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-netns\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-ovn\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqpm\" (UniqueName: \"kubernetes.io/projected/d4acf9b8-e56f-4958-9742-7303e972ae20-kube-api-access-qqqpm\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.777739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-kubelet-config\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.777739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-etc-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-kubelet\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.777898 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc620-2f0c-4060-b76e-2b2365e3621f-tmp-dir\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.777898 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-kubelet\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777898 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-slash\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777898 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-log-socket\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777898 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-run\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.777898 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-systemd-units\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.777898 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-sys\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-etc-selinux\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.777963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gd4\" (UniqueName: \"kubernetes.io/projected/82127ba1-4bef-43cd-8550-719a4402fcab-kube-api-access-54gd4\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-binary-copy\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778010 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8bc620-2f0c-4060-b76e-2b2365e3621f-hosts-file\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-lib-modules\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-device-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-multus\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778223 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-systemd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-multus-certs\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.778272 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrg4j\" (UniqueName: \"kubernetes.io/projected/0bbee04d-3f06-4ce5-8020-4d52b9715887-kube-api-access-vrg4j\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:35.778976 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778170 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-netns\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.778976 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778496 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-multus\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.778976 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778538 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-host\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.778976 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.778800 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-log-socket\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779225 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779076 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc620-2f0c-4060-b76e-2b2365e3621f-tmp-dir\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.779225 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779088 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-device-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.779225 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779129 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-lib-modules\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.779225 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779158 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779225 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779193 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-kubelet-config\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.779225 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779206 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-kubelet\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779225 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779210 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.779580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779248 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-etc-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779260 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-slash\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779294 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-kubelet\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.779580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779303 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-multus-certs\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.779580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779372 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-sys\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.779580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.779580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779501 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8bc620-2f0c-4060-b76e-2b2365e3621f-hosts-file\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.779580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-var-lib-kubelet\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-socket-dir-parent\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-kubernetes\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-var-lib-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-node-log\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779808 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-script-lib\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.779991 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.779910 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-binary-copy\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-system-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780055 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-node-log\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780083 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-cnibin\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780106 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-var-lib-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780150 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-ovn\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-netd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780191 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-netns\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-config\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-env-overrides\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-host\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780280 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-socket-dir-parent\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d05019b-efc6-444a-986d-6f63420ba3ff-konnectivity-ca\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780362 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-conf\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780395 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-host\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780410 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-var-lib-kubelet\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.780484 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780468 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780511 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-systemd-units\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780560 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-run\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780792 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-kubernetes\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780843 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-netd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780865 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-env-overrides\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-tuned\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7906b48-2326-4074-a2a4-a62654c2b785-iptables-alerter-script\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.780947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-sys-fs\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-bin\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cni-binary-copy\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-conf-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781147 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-script-lib\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781166 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781213 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-bin\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781236 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-system-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.781369 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.781356 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:35.782390 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.781494 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs podName:0bbee04d-3f06-4ce5-8020-4d52b9715887 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:36.281452377 +0000 UTC m=+3.164190937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs") pod "network-metrics-daemon-hchmm" (UID: "0bbee04d-3f06-4ce5-8020-4d52b9715887") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:35.782390 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781576 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-etc-selinux\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.782390 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.781683 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d05019b-efc6-444a-986d-6f63420ba3ff-konnectivity-ca\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:35.782390 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.782382 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-config\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.782606 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.782535 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-conf-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.782606 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.782575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-serviceca\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.782707 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.782621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.782757 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.782726 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-sys-fs\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.782997 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.782832 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-cnibin\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783095 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krrq\" (UniqueName: \"kubernetes.io/projected/3b8bc620-2f0c-4060-b76e-2b2365e3621f-kube-api-access-6krrq\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vph5m\" (UniqueName: \"kubernetes.io/projected/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kube-api-access-vph5m\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-etc-kubernetes\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-system-cni-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-dbus\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783478 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-modprobe-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cnibin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-bin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7906b48-2326-4074-a2a4-a62654c2b785-host-slash\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.783842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-bin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784046 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-dbus\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:35.786062 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7z4\" (UniqueName: \"kubernetes.io/projected/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-kube-api-access-dx7z4\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784091 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-modprobe-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784228 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7906b48-2326-4074-a2a4-a62654c2b785-host-slash\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784402 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7906b48-2326-4074-a2a4-a62654c2b785-iptables-alerter-script\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784442 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-system-cni-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysconfig\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784493 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cnibin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-systemd\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-etc-kubernetes\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.784585 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-tmp\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-socket-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-daemon-config\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784886 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysconfig\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.784985 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-systemd\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.785113 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret podName:8e9cf883-7d59-46f0-b457-f84c7c678251 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:36.285081212 +0000 UTC m=+3.167819754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret") pod "global-pull-secret-syncer-qg2lp" (UID: "8e9cf883-7d59-46f0-b457-f84c7c678251") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:35.787739 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.785181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chz5j\" (UniqueName: \"kubernetes.io/projected/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-kube-api-access-chz5j\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.789172 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.785245 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cni-binary-copy\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.789172 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.785514 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-socket-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.789172 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.785900 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-daemon-config\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.789172 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.787108 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.789985 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.789850 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-serviceca\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.789985 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.789872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-tuned\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.790353 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.790178 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82127ba1-4bef-43cd-8550-719a4402fcab-ovn-node-metrics-cert\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.790615 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.790453 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d05019b-efc6-444a-986d-6f63420ba3ff-agent-certs\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:35.793377 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.793182 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-tmp\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.807064 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.805850 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:35.807064 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.805919 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:35.807064 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.805948 2576 projected.go:196] Error preparing data for projected volume kube-api-access-8pmv5 for pod openshift-network-diagnostics/network-check-target-xpbwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:35.807064 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:35.806122 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5 podName:7e4312f3-14a2-49b4-a6f1-4484d94a2217 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:36.306084342 +0000 UTC m=+3.188822894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8pmv5" (UniqueName: "kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5") pod "network-check-target-xpbwl" (UID: "7e4312f3-14a2-49b4-a6f1-4484d94a2217") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:35.807064 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.806429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrg4j\" (UniqueName: \"kubernetes.io/projected/0bbee04d-3f06-4ce5-8020-4d52b9715887-kube-api-access-vrg4j\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:35.807064 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.806433 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4qt\" (UniqueName: \"kubernetes.io/projected/d7906b48-2326-4074-a2a4-a62654c2b785-kube-api-access-gn4qt\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:35.809030 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.808751 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vph5m\" (UniqueName: \"kubernetes.io/projected/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kube-api-access-vph5m\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:35.812101 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.812032 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqpm\" (UniqueName: \"kubernetes.io/projected/d4acf9b8-e56f-4958-9742-7303e972ae20-kube-api-access-qqqpm\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.812411 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.812033 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7kw\" (UniqueName: \"kubernetes.io/projected/5f811312-1c2a-4099-9cbe-efe1c5063b66-kube-api-access-qw7kw\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.812847 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.812804 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chz5j\" (UniqueName: \"kubernetes.io/projected/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-kube-api-access-chz5j\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:00:35.813076 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.812812 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krrq\" (UniqueName: \"kubernetes.io/projected/3b8bc620-2f0c-4060-b76e-2b2365e3621f-kube-api-access-6krrq\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:35.814218 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.814051 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gd4\" (UniqueName: \"kubernetes.io/projected/82127ba1-4bef-43cd-8550-719a4402fcab-kube-api-access-54gd4\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:35.821542 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.821477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7z4\" (UniqueName: \"kubernetes.io/projected/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-kube-api-access-dx7z4\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.936078 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.936011 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:00:35.951817 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.951317 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:00:35.966883 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:35.966809 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f811312_1c2a_4099_9cbe_efe1c5063b66.slice/crio-2610572b668afe1c5d57da59294aefd21c078ef9b6919bcaef77b48b40636deb WatchSource:0}: Error finding container 2610572b668afe1c5d57da59294aefd21c078ef9b6919bcaef77b48b40636deb: Status 404 returned error can't find the container with id 2610572b668afe1c5d57da59294aefd21c078ef9b6919bcaef77b48b40636deb Oct 30 18:00:35.996413 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.996349 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:00:35.997583 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:35.997512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:00:36.013947 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.013893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:00:36.020619 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:36.020498 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4acf9b8_e56f_4958_9742_7303e972ae20.slice/crio-543d5806760cc4d4ff67ee214ae680634507a34462f7569d91186dab578caa5d WatchSource:0}: Error finding container 543d5806760cc4d4ff67ee214ae680634507a34462f7569d91186dab578caa5d: Status 404 returned error can't find the container with id 543d5806760cc4d4ff67ee214ae680634507a34462f7569d91186dab578caa5d Oct 30 18:00:36.022615 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.022109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:36.040823 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:36.040731 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca1d631_c3c1_4e4f_910c_76a1310a327c.slice/crio-ff6b759ead661c112c41275ddf4276b3b0cf22b4715d6e79fb0799755a93f4ca WatchSource:0}: Error finding container ff6b759ead661c112c41275ddf4276b3b0cf22b4715d6e79fb0799755a93f4ca: Status 404 returned error can't find the container with id ff6b759ead661c112c41275ddf4276b3b0cf22b4715d6e79fb0799755a93f4ca Oct 30 18:00:36.065606 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.065544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n97j8" Oct 30 18:00:36.069405 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.069332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:36.081094 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.080583 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sq5cb" Oct 30 18:00:36.097801 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:00:36.097725 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8bc620_2f0c_4060_b76e_2b2365e3621f.slice/crio-51c430ef6903b5afe43d542f5ea3b5cf6376b8aea0ed810bdf98e6f784c019df WatchSource:0}: Error finding container 51c430ef6903b5afe43d542f5ea3b5cf6376b8aea0ed810bdf98e6f784c019df: Status 404 returned error can't find the container with id 51c430ef6903b5afe43d542f5ea3b5cf6376b8aea0ed810bdf98e6f784c019df Oct 30 18:00:36.292655 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.292507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:36.292655 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.292612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:36.294095 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:36.292751 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:36.294095 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:36.292827 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:36.294095 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:36.292893 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs podName:0bbee04d-3f06-4ce5-8020-4d52b9715887 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:37.292861987 +0000 UTC m=+4.175600521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs") pod "network-metrics-daemon-hchmm" (UID: "0bbee04d-3f06-4ce5-8020-4d52b9715887") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:36.294095 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:36.292918 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret podName:8e9cf883-7d59-46f0-b457-f84c7c678251 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:37.292910426 +0000 UTC m=+4.175648958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret") pod "global-pull-secret-syncer-qg2lp" (UID: "8e9cf883-7d59-46f0-b457-f84c7c678251") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:36.394282 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.394210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:36.394606 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:36.394520 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:36.394606 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:36.394559 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:36.394606 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:36.394576 2576 projected.go:196] Error preparing data for projected volume kube-api-access-8pmv5 for pod openshift-network-diagnostics/network-check-target-xpbwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:36.394728 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:36.394663 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5 podName:7e4312f3-14a2-49b4-a6f1-4484d94a2217 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:37.394640138 +0000 UTC m=+4.277378687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pmv5" (UniqueName: "kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5") pod "network-check-target-xpbwl" (UID: "7e4312f3-14a2-49b4-a6f1-4484d94a2217") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:36.635172 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.634824 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:35 +0000 UTC" deadline="2028-05-29 16:00:01.47761318 +0000 UTC" Oct 30 18:00:36.635172 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.634934 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13845h59m24.842685173s" Oct 30 18:00:36.719100 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.718297 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6w92c" event={"ID":"5d05019b-efc6-444a-986d-6f63420ba3ff","Type":"ContainerStarted","Data":"96906cdfd91d34ffc8ba2c2349383c5d85b2bc9d27cef530921e2c112027f025"} Oct 30 18:00:36.726129 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.725756 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"ad1c70e922ad2d188d1e2a2b376e1dd72574b2fa11f4dcc332c49a174339416f"} Oct 30 18:00:36.731112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.730885 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n97j8" event={"ID":"3b8bc620-2f0c-4060-b76e-2b2365e3621f","Type":"ContainerStarted","Data":"51c430ef6903b5afe43d542f5ea3b5cf6376b8aea0ed810bdf98e6f784c019df"} Oct 30 18:00:36.761417 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.761262 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" event={"ID":"8ca1d631-c3c1-4e4f-910c-76a1310a327c","Type":"ContainerStarted","Data":"ff6b759ead661c112c41275ddf4276b3b0cf22b4715d6e79fb0799755a93f4ca"} Oct 30 18:00:36.769564 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.769436 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fl2ts" event={"ID":"d7906b48-2326-4074-a2a4-a62654c2b785","Type":"ContainerStarted","Data":"fc0d8ffb48b2edb299cb70782535f4139f77ed4fef2addac6865033eb11f0f3d"} Oct 30 18:00:36.782573 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.782393 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerStarted","Data":"543d5806760cc4d4ff67ee214ae680634507a34462f7569d91186dab578caa5d"} Oct 30 18:00:36.790643 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.790418 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l6vjs" event={"ID":"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2","Type":"ContainerStarted","Data":"013f758ad4f0ef581b673c828693c4f15f85ae610e932510dcfc200c774ef0f0"} Oct 30 18:00:36.808184 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.808112 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" event={"ID":"5f811312-1c2a-4099-9cbe-efe1c5063b66","Type":"ContainerStarted","Data":"2610572b668afe1c5d57da59294aefd21c078ef9b6919bcaef77b48b40636deb"} Oct 30 18:00:36.828751 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:36.828202 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sq5cb" event={"ID":"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b","Type":"ContainerStarted","Data":"c16fe87eb3e0bcbd122feeb1530c976da6144525f3645851b01a6029bafcc5a1"} Oct 30 18:00:37.306088 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:37.304790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:37.306088 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:37.304901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:37.306088 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.305130 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:37.306088 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.305343 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs podName:0bbee04d-3f06-4ce5-8020-4d52b9715887 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.305299837 +0000 UTC m=+6.188038737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs") pod "network-metrics-daemon-hchmm" (UID: "0bbee04d-3f06-4ce5-8020-4d52b9715887") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:37.306088 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.305137 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:37.306753 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.306147 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret podName:8e9cf883-7d59-46f0-b457-f84c7c678251 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.306114182 +0000 UTC m=+6.188852731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret") pod "global-pull-secret-syncer-qg2lp" (UID: "8e9cf883-7d59-46f0-b457-f84c7c678251") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:37.407092 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:37.405860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:37.407092 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.406212 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:37.407092 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.406250 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:37.407092 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.406273 2576 projected.go:196] Error preparing data for projected volume kube-api-access-8pmv5 for pod openshift-network-diagnostics/network-check-target-xpbwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:37.407092 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.406397 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5 podName:7e4312f3-14a2-49b4-a6f1-4484d94a2217 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:39.406363176 +0000 UTC m=+6.289101738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pmv5" (UniqueName: "kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5") pod "network-check-target-xpbwl" (UID: "7e4312f3-14a2-49b4-a6f1-4484d94a2217") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:37.636401 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:37.636159 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:35 +0000 UTC" deadline="2028-06-25 18:33:44.592138176 +0000 UTC" Oct 30 18:00:37.636401 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:37.636236 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14496h33m6.955908044s" Oct 30 18:00:37.677276 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:37.673824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:37.677276 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.674644 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:37.677276 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:37.674331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:37.677276 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.674828 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:37.677276 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:37.674260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:37.677276 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:37.674960 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:39.325239 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.325173 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:39.326068 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.325263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:39.326068 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.325576 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:39.326068 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.325727 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret podName:8e9cf883-7d59-46f0-b457-f84c7c678251 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:43.325692008 +0000 UTC m=+10.208430559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret") pod "global-pull-secret-syncer-qg2lp" (UID: "8e9cf883-7d59-46f0-b457-f84c7c678251") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:39.326308 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.326179 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:39.326374 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.326303 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs podName:0bbee04d-3f06-4ce5-8020-4d52b9715887 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:43.326275093 +0000 UTC m=+10.209013630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs") pod "network-metrics-daemon-hchmm" (UID: "0bbee04d-3f06-4ce5-8020-4d52b9715887") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:39.427083 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.426907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:39.427452 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.427273 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:39.427452 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.427310 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:39.427452 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.427328 2576 projected.go:196] Error preparing data for projected volume kube-api-access-8pmv5 for pod openshift-network-diagnostics/network-check-target-xpbwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:39.427452 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.427425 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5 podName:7e4312f3-14a2-49b4-a6f1-4484d94a2217 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:43.427398848 +0000 UTC m=+10.310137386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pmv5" (UniqueName: "kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5") pod "network-check-target-xpbwl" (UID: "7e4312f3-14a2-49b4-a6f1-4484d94a2217") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:39.676574 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.675688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:39.676574 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.675998 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:39.676574 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.676154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:39.676574 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.676294 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:39.676574 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.676402 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:39.676574 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:39.676499 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:39.884025 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.883803 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" event={"ID":"05806b328c870eba3d65f2b11ca3bf01","Type":"ContainerStarted","Data":"fbe6b579808550835f3e59759685199b7a201c7007d1cab5a714c4ed172dbc1f"} Oct 30 18:00:39.898796 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.898721 2576 generic.go:356] "Generic (PLEG): container finished" podID="9bfefcd36786ca02a8752b4bece34acc" containerID="10db0fe1a1ec6c19b84ea7d63467105f64b720c15d8778bde9304338ea9fd028" exitCode=0 Oct 30 18:00:39.899134 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.899010 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" event={"ID":"9bfefcd36786ca02a8752b4bece34acc","Type":"ContainerDied","Data":"10db0fe1a1ec6c19b84ea7d63467105f64b720c15d8778bde9304338ea9fd028"} Oct 30 18:00:39.922084 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.921903 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerStarted","Data":"a99731a377b7328db31b8393afc3b127794ba1dda7d3ef4cb0edd6c7cf173dd1"} Oct 30 18:00:39.991051 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:39.990913 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" podStartSLOduration=4.99087607 podStartE2EDuration="4.99087607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:39.943018478 +0000 UTC m=+6.825757032" watchObservedRunningTime="2026-10-30 18:00:39.99087607 +0000 UTC m=+6.873614625" Oct 30 18:00:40.937741 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:40.937597 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="a99731a377b7328db31b8393afc3b127794ba1dda7d3ef4cb0edd6c7cf173dd1" exitCode=0 Oct 30 18:00:40.938699 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:40.937872 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerDied","Data":"a99731a377b7328db31b8393afc3b127794ba1dda7d3ef4cb0edd6c7cf173dd1"} Oct 30 18:00:40.954333 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:40.954257 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" event={"ID":"9bfefcd36786ca02a8752b4bece34acc","Type":"ContainerStarted","Data":"70a7fcc62bf6514a3536b602e13d70063402bb8d9dc22351c70a74e8d50d0e28"} Oct 30 18:00:40.997095 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:40.996886 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" podStartSLOduration=5.996850045 podStartE2EDuration="5.996850045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:40.996493103 +0000 UTC m=+7.879231704" watchObservedRunningTime="2026-10-30 18:00:40.996850045 +0000 UTC m=+7.879588600" Oct 30 18:00:41.674281 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:41.674180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:41.674587 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:41.674444 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:41.675129 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:41.675099 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:41.675317 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:41.675278 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:41.675598 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:41.675563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:41.675756 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:41.675722 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:41.962548 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:41.962477 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6w92c" event={"ID":"5d05019b-efc6-444a-986d-6f63420ba3ff","Type":"ContainerStarted","Data":"30d8e222859d57397ea6be437811aef3d32ccc4e1b7b7f3c285eddc7d8a5fba0"} Oct 30 18:00:42.007913 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:42.006599 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6w92c" podStartSLOduration=2.757212917 podStartE2EDuration="8.00656637s" totalImagesPullingTime="5.249353453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:42.00590949 +0000 UTC m=+8.888648044" watchObservedRunningTime="2026-10-30 18:00:42.00656637 +0000 UTC m=+8.889304945" Oct 30 18:00:43.374186 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:43.374128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:43.374921 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:43.374214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:43.374921 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.374457 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:43.374921 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.374576 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret podName:8e9cf883-7d59-46f0-b457-f84c7c678251 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:51.374548339 +0000 UTC m=+18.257286889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret") pod "global-pull-secret-syncer-qg2lp" (UID: "8e9cf883-7d59-46f0-b457-f84c7c678251") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:43.375310 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.375276 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:43.375416 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.375391 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs podName:0bbee04d-3f06-4ce5-8020-4d52b9715887 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:51.375364698 +0000 UTC m=+18.258103279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs") pod "network-metrics-daemon-hchmm" (UID: "0bbee04d-3f06-4ce5-8020-4d52b9715887") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:43.475254 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:43.475183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:43.475584 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.475498 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:43.475584 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.475526 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:43.475584 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.475542 2576 projected.go:196] Error preparing data for projected volume kube-api-access-8pmv5 for pod openshift-network-diagnostics/network-check-target-xpbwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:43.475744 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.475638 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5 podName:7e4312f3-14a2-49b4-a6f1-4484d94a2217 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:51.475612396 +0000 UTC m=+18.358350950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pmv5" (UniqueName: "kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5") pod "network-check-target-xpbwl" (UID: "7e4312f3-14a2-49b4-a6f1-4484d94a2217") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:43.679649 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:43.679182 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:43.679649 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.679492 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:43.681512 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:43.681123 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:43.681512 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.681257 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:43.681512 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:43.681352 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:43.681512 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:43.681401 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:43.977288 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:43.977119 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" event={"ID":"8ca1d631-c3c1-4e4f-910c-76a1310a327c","Type":"ContainerStarted","Data":"da9590196a91de98ada79d279b8fb81837557b2dc0f96964bf70206e60530925"} Oct 30 18:00:43.984536 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:43.984454 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l6vjs" event={"ID":"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2","Type":"ContainerStarted","Data":"3a1a8392f1b53db22b72156f0a4dfc764afefbc32a7cd3d9f3fb7d9955861ae8"} Oct 30 18:00:44.031112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:44.030979 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l6vjs" podStartSLOduration=3.271073249 podStartE2EDuration="10.030912663s" totalImagesPullingTime="6.759839414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:44.03039348 +0000 UTC m=+10.913132064" watchObservedRunningTime="2026-10-30 18:00:44.030912663 +0000 UTC m=+10.913651217" Oct 30 18:00:44.159402 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:44.159292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:44.280562 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:44.280303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:44.388508 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:44.388361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:44.590996 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:44.590770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:44.693723 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:44.693516 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:44.795766 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:44.795595 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:45.354407 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:45.354261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:45.459222 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:45.459158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:45.604639 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:45.603224 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:45.606109 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:45.606047 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:45.676001 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:45.675089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:45.676001 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:45.675133 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:45.676001 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:45.675089 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:45.676001 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:45.675310 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:45.676001 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:45.675481 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:45.676001 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:45.675608 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:45.884205 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:45.883788 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:00:46.005451 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.005380 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" event={"ID":"8ca1d631-c3c1-4e4f-910c-76a1310a327c","Type":"ContainerStarted","Data":"ae9e6c7e7e7d269927fff711310404e5f1550547fd44a5e658b7ac6076a2b18d"} Oct 30 18:00:46.018586 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.018520 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerStarted","Data":"f0af9f0c40b697a4949d516b721801d2e5a3d9f1fb14a30de31d6e9fc6ea596a"} Oct 30 18:00:46.018928 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.018625 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:46.018928 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.018692 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:00:46.266839 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.266693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:46.649392 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.648276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:46.683746 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.683530 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:00:45.884196072Z","UUID":"1e67b52a-9bbf-4c0d-a062-f3edfd7c3d8a","Handler":null,"Name":"","Endpoint":""} Oct 30 18:00:46.686784 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.686733 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:00:46.686784 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.686790 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:00:46.820853 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.820774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:46.922729 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:46.922588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:47.025915 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.025684 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n97j8" event={"ID":"3b8bc620-2f0c-4060-b76e-2b2365e3621f","Type":"ContainerStarted","Data":"69ed1801aa358d19cf6ee323dd45c81b13316bb3526ca9073668b67f702b802c"} Oct 30 18:00:47.032492 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.032402 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="f0af9f0c40b697a4949d516b721801d2e5a3d9f1fb14a30de31d6e9fc6ea596a" exitCode=0 Oct 30 18:00:47.033791 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.033725 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerDied","Data":"f0af9f0c40b697a4949d516b721801d2e5a3d9f1fb14a30de31d6e9fc6ea596a"} Oct 30 18:00:47.039389 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.039321 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sq5cb" event={"ID":"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b","Type":"ContainerStarted","Data":"87bb400e3ed2374768094e855b31c62a6021416732b4acac5baded65b085a9a0"} Oct 30 18:00:47.156646 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.156532 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-n97j8" podStartSLOduration=2.8868981959999997 podStartE2EDuration="13.156504852s" totalImagesPullingTime="10.269606656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:47.106484692 +0000 UTC m=+13.989223248" watchObservedRunningTime="2026-10-30 18:00:47.156504852 +0000 UTC m=+14.039243409" Oct 30 18:00:47.203913 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.203793 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-sq5cb" podStartSLOduration=3.3528167509999998 podStartE2EDuration="13.203758204s" totalImagesPullingTime="9.850941453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:47.203274109 +0000 UTC m=+14.086012665" watchObservedRunningTime="2026-10-30 18:00:47.203758204 +0000 UTC m=+14.086496759" Oct 30 18:00:47.347737 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.347644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:47.675734 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.675557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:47.676816 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:47.675807 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:47.676816 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.676524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:47.676816 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:47.676680 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:47.676816 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:47.676807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:47.677037 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:47.676910 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:48.050849 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:48.050760 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerStarted","Data":"f4d04c3962bbc760c948f7c24a7d21e5c7ae6fd6e84cc05e63afb858c016f074"} Oct 30 18:00:48.056426 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:48.056315 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" event={"ID":"8ca1d631-c3c1-4e4f-910c-76a1310a327c","Type":"ContainerStarted","Data":"00c2b3144165a888b215ec9a162130cfb0e00981f547a2b4d17208715f6b76bb"} Oct 30 18:00:48.150978 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:48.150825 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" podStartSLOduration=3.773587065 podStartE2EDuration="14.150780219s" totalImagesPullingTime="10.377193154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:48.148870742 +0000 UTC m=+15.031609302" watchObservedRunningTime="2026-10-30 18:00:48.150780219 +0000 UTC m=+15.033518777" Oct 30 18:00:48.156336 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:48.154328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:48.542319 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:48.542264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:48.757849 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:48.757682 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:48.855986 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:48.855802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:48.956433 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:48.956362 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:49.065025 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.064950 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="f4d04c3962bbc760c948f7c24a7d21e5c7ae6fd6e84cc05e63afb858c016f074" exitCode=0 Oct 30 18:00:49.065320 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.065150 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerDied","Data":"f4d04c3962bbc760c948f7c24a7d21e5c7ae6fd6e84cc05e63afb858c016f074"} Oct 30 18:00:49.068910 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.068711 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" event={"ID":"5f811312-1c2a-4099-9cbe-efe1c5063b66","Type":"ContainerStarted","Data":"f7c453bd17298a83cd8090762546b2363b118877cca3e326286ad68be6dc0aa2"} Oct 30 18:00:49.160865 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.160667 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" podStartSLOduration=2.587083505 podStartE2EDuration="15.160636705s" totalImagesPullingTime="12.5735532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:49.159309936 +0000 UTC m=+16.042048490" watchObservedRunningTime="2026-10-30 18:00:49.160636705 +0000 UTC m=+16.043375259" Oct 30 18:00:49.349214 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.349130 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:49.650433 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.650259 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:49.673902 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.673836 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:49.673902 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.673920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:49.674327 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:49.674126 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:49.674327 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:49.674207 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:49.674441 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:49.674339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:49.674488 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:49.674453 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:50.075625 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.075107 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fl2ts" event={"ID":"d7906b48-2326-4074-a2a4-a62654c2b785","Type":"ContainerStarted","Data":"fe391b7b7415139ed120931a19bfb37d28b717fc457041d0591252fa2bb9ba1b"} Oct 30 18:00:50.081367 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.081303 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="96d28d0887239ac5b3771203150b689afccdad44f9752eb21ec23f7f06768068" exitCode=0 Oct 30 18:00:50.082078 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.082021 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerDied","Data":"96d28d0887239ac5b3771203150b689afccdad44f9752eb21ec23f7f06768068"} Oct 30 18:00:50.119125 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.119017 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fl2ts" podStartSLOduration=5.404485899 podStartE2EDuration="16.118980858s" totalImagesPullingTime="10.714494959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:50.117893589 +0000 UTC m=+17.000632161" watchObservedRunningTime="2026-10-30 18:00:50.118980858 +0000 UTC m=+17.001719609" Oct 30 18:00:50.162553 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.162336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:50.263838 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.263744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:50.540445 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.540290 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:50.637551 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.637483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:50.753175 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.753008 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n97j8_3b8bc620-2f0c-4060-b76e-2b2365e3621f/dns-node-resolver/0.log" Oct 30 18:00:50.753488 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.753341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:50.941471 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:50.941310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:51.141815 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.141750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:51.375143 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.374968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:51.375449 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.375275 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:51.375449 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.375388 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret podName:8e9cf883-7d59-46f0-b457-f84c7c678251 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:07.375360702 +0000 UTC m=+34.258099234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret") pod "global-pull-secret-syncer-qg2lp" (UID: "8e9cf883-7d59-46f0-b457-f84c7c678251") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:51.428576 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.428495 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:51.475918 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.475836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:51.476233 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.475974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:51.476233 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.476196 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:51.476233 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.476229 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:51.476383 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.476248 2576 projected.go:196] Error preparing data for projected volume kube-api-access-8pmv5 for pod openshift-network-diagnostics/network-check-target-xpbwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:51.476383 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.476330 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5 podName:7e4312f3-14a2-49b4-a6f1-4484d94a2217 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:07.476312766 +0000 UTC m=+34.359051297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pmv5" (UniqueName: "kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5") pod "network-check-target-xpbwl" (UID: "7e4312f3-14a2-49b4-a6f1-4484d94a2217") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:51.476997 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.476962 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:51.477172 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.477038 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs podName:0bbee04d-3f06-4ce5-8020-4d52b9715887 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:07.477020672 +0000 UTC m=+34.359759217 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs") pod "network-metrics-daemon-hchmm" (UID: "0bbee04d-3f06-4ce5-8020-4d52b9715887") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:51.542826 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.542740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:51.673447 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.673275 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:51.673447 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.673284 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:51.673844 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.673485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:51.673844 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.673502 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:51.673844 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.673664 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:51.674026 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:51.673839 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:51.753589 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.753533 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l6vjs_c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2/node-ca/0.log" Oct 30 18:00:51.809919 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.809856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:51.911533 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:51.911451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:52.096144 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:52.096072 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerStarted","Data":"ba97567dd1afd3c4d2a3b9881f911aa35f4c93cb0bb1957dbe709f0e74bd4c6b"} Oct 30 18:00:52.331556 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:52.331407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:52.424816 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:52.424745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:52.554267 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:52.554190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:52.745561 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:52.745402 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:52.937377 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:52.937305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:53.046480 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.046337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:53.105534 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.105460 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="ba97567dd1afd3c4d2a3b9881f911aa35f4c93cb0bb1957dbe709f0e74bd4c6b" exitCode=0 Oct 30 18:00:53.105534 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.105535 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerDied","Data":"ba97567dd1afd3c4d2a3b9881f911aa35f4c93cb0bb1957dbe709f0e74bd4c6b"} Oct 30 18:00:53.147424 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.147369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:53.247367 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.247306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:53.433206 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.433134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:53.530169 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.530108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:53.674263 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.674201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:53.674263 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.674254 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:53.674669 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:53.674458 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:53.674669 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:53.674453 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:53.674669 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:53.674585 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:53.674833 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:53.674725 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:54.118145 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.117538 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="10e29837eeda3ba13ab7b3a11c102287a128a9e2b509629e973205e97d81d371" exitCode=0 Oct 30 18:00:54.118421 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.117634 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerDied","Data":"10e29837eeda3ba13ab7b3a11c102287a128a9e2b509629e973205e97d81d371"} Oct 30 18:00:54.126042 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.125962 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"0ecdb43fe35695254f1ca0abb75d2c7868d465d7ac7151c98845bc2ac1e6487c"} Oct 30 18:00:54.126319 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.126109 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"738beaa397c99b145dad2ee269d8b7bbf7f66ba12a32b7b63bfc107c16d525d3"} Oct 30 18:00:54.126319 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.126156 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"5389f260ad9f345bcb26af0072fd50596ef6cdae63d41716b13de23faaa5dbc1"} Oct 30 18:00:54.126319 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.126204 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"956e573923b18816b56ace605a47265bf820f513557ba376e4cd4df1549a7f13"} Oct 30 18:00:54.126319 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.126251 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"f48acdab00a8fb353045961005772e11c0d0327536aaf788ef846e0aff439295"} Oct 30 18:00:54.126319 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.126309 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"4e356659971e520c10baa06dea64ce3ac14d9c5361da3d09c2b1c1f1eb33180f"} Oct 30 18:00:54.142905 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.142627 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:54.429034 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.428824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:54.563618 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.563551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:54.666393 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.666328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:54.766923 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.766782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:54.944287 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:54.944226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:55.136785 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:55.136597 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" event={"ID":"d4acf9b8-e56f-4958-9742-7303e972ae20","Type":"ContainerStarted","Data":"728530cf94274a5ecfa3bfef97913d11dcdf3360c588a924d38f5e69a2afc023"} Oct 30 18:00:55.185855 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:55.185715 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gdxmg" podStartSLOduration=8.885471633 podStartE2EDuration="21.185685222s" totalImagesPullingTime="12.300213589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:55.184991444 +0000 UTC m=+22.067729995" watchObservedRunningTime="2026-10-30 18:00:55.185685222 +0000 UTC m=+22.068423775" Oct 30 18:00:55.339093 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:55.339036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:55.672985 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:55.672910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:55.673681 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:55.672910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:55.673681 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:55.673121 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:55.673681 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:55.672952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:55.673681 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:55.673196 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:55.673681 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:55.673311 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:56.143580 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:56.143421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:56.535121 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:56.534916 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:56.741170 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:56.741115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:56.948257 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:56.948109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:57.122146 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.122096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:57.149623 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.149556 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"2797be03be9d18b0b652858de55f757db3d2ec2ecd5e557abb4967cd131c6e5f"} Oct 30 18:00:57.222845 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.222660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:57.323369 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.323292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:57.463845 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.463790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:57.565715 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.565530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:57.664867 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.664800 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:57.673841 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.673787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:57.673841 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.673824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:57.674098 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.673787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:57.674156 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:57.674125 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:00:57.674215 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:57.674185 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:57.674253 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:57.674238 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:57.942659 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:57.942481 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:59.164310 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:59.163914 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" event={"ID":"82127ba1-4bef-43cd-8550-719a4402fcab","Type":"ContainerStarted","Data":"1254f6a46f0585d633c89b83248987e957c4719fe4f07e10065375a0bfd9a3f0"} Oct 30 18:00:59.165323 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:59.164436 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:59.165323 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:59.164488 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:59.192874 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:59.192827 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:00:59.256544 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:59.256424 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" podStartSLOduration=8.001043053 podStartE2EDuration="25.256402664s" totalImagesPullingTime="17.255359611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:59.242096094 +0000 UTC m=+26.124834647" watchObservedRunningTime="2026-10-30 18:00:59.256402664 +0000 UTC m=+26.139141271" Oct 30 18:00:59.673979 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:59.673875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:00:59.673979 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:59.673960 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:00:59.673979 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:00:59.673880 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:00:59.674336 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:59.674125 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:00:59.674336 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:59.674269 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:00:59.674445 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:00:59.674377 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:00.168539 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:00.168484 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:00.194404 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:00.194361 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:01.673113 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:01.673061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:01.673962 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:01.673061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:01.673962 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:01.673247 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:01:01.673962 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:01.673077 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:01.673962 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:01.673281 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:01:01.673962 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:01.673394 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:03.675073 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:03.674996 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:03.675073 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:03.675061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:03.675888 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:03.675161 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:03.675888 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:03.675218 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:01:03.675888 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:03.675259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:03.675888 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:03.675371 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:01:05.592166 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:05.592105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:05.673979 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:05.673909 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:05.674323 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:05.673903 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:05.674323 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:05.674102 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:01:05.674323 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:05.674131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:05.674323 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:05.674254 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:05.680311 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:05.674333 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:01:05.681885 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:05.681828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:07.430105 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:07.429997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:07.430782 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.430207 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:01:07.430782 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.430304 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret podName:8e9cf883-7d59-46f0-b457-f84c7c678251 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:39.43027755 +0000 UTC m=+66.313016097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret") pod "global-pull-secret-syncer-qg2lp" (UID: "8e9cf883-7d59-46f0-b457-f84c7c678251") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:01:07.530689 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:07.530626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:07.531056 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:07.530741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:07.531056 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.530867 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:01:07.531056 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.530883 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:01:07.531056 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.530898 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:01:07.531056 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.530914 2576 projected.go:196] Error preparing data for projected volume kube-api-access-8pmv5 for pod openshift-network-diagnostics/network-check-target-xpbwl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:01:07.531056 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.531012 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5 podName:7e4312f3-14a2-49b4-a6f1-4484d94a2217 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:39.530985709 +0000 UTC m=+66.413724243 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8pmv5" (UniqueName: "kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5") pod "network-check-target-xpbwl" (UID: "7e4312f3-14a2-49b4-a6f1-4484d94a2217") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:01:07.531056 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.531045 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs podName:0bbee04d-3f06-4ce5-8020-4d52b9715887 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:39.531035623 +0000 UTC m=+66.413774156 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs") pod "network-metrics-daemon-hchmm" (UID: "0bbee04d-3f06-4ce5-8020-4d52b9715887") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:01:07.673681 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:07.673613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:07.674028 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:07.673613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:07.674028 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:07.673613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:07.674028 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.673824 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:01:07.674028 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.673925 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:07.674028 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:07.674014 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:01:08.904450 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:08.904372 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:09.026988 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.026879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:09.227418 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.227211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:09.661687 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.661604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:09.673626 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.673548 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:09.674015 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.673581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:09.674015 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.673778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:09.674156 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:09.673993 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:01:09.674156 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:09.674034 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:01:09.674156 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:09.674059 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:09.902678 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.902605 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qg2lp"] Oct 30 18:01:09.906778 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.906449 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hchmm"] Oct 30 18:01:09.907399 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:09.907151 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpbwl"] Oct 30 18:01:10.200329 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:10.200179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:10.200329 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:10.200179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:10.200329 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:10.200179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:10.200693 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:10.200388 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:01:10.200693 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:10.200480 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:10.200693 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:10.200568 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:01:10.639574 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:10.639504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:10.739383 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:10.738950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:11.673428 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:11.673361 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:11.674184 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:11.673362 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:11.674184 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:11.673557 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:01:11.674184 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:11.673576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:11.674184 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:11.673708 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:11.674184 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:11.673837 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:01:12.672250 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:12.672176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:12.777458 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:12.777379 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:12.898596 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:12.898428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:13.676336 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:13.676272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:13.676336 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:13.676334 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:13.676795 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:13.676303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:13.676795 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:13.676511 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hchmm" podUID="0bbee04d-3f06-4ce5-8020-4d52b9715887" Oct 30 18:01:13.676795 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:13.676628 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qg2lp" podUID="8e9cf883-7d59-46f0-b457-f84c7c678251" Oct 30 18:01:13.676795 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:13.676731 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpbwl" podUID="7e4312f3-14a2-49b4-a6f1-4484d94a2217" Oct 30 18:01:13.889772 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:13.889700 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:14.647599 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.647537 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:14.733964 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.733866 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:14.801971 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.801799 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeReady" Oct 30 18:01:14.802290 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.802093 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 18:01:14.863189 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.863120 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6mcgq"] Oct 30 18:01:14.867366 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.867306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:14.867840 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.867792 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9l55k"] Oct 30 18:01:14.872149 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.872107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:14.888985 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.888882 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6mcgq"] Oct 30 18:01:14.892154 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.891575 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:01:14.894182 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.894123 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:01:14.894472 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.894263 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:01:14.894472 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.894315 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:01:14.894472 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.894406 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" Oct 30 18:01:14.894472 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.894429 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:01:14.894890 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.894854 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" Oct 30 18:01:14.896275 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.896231 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9l55k"] Oct 30 18:01:14.969390 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.969321 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x6tdb"] Oct 30 18:01:14.973572 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.973524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:14.992394 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.992319 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:01:14.992769 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.992608 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:01:14.992769 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.992666 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" Oct 30 18:01:14.992769 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.992738 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:01:14.993149 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.993113 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:01:14.997113 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:14.997014 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x6tdb"] Oct 30 18:01:15.004890 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.004832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cd5923-8633-4b24-8310-e06ac0a2de68-config-volume\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.004890 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.004891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhg7\" (UniqueName: \"kubernetes.io/projected/b1cd5923-8633-4b24-8310-e06ac0a2de68-kube-api-access-nlhg7\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.005307 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.005101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vz24\" (UniqueName: \"kubernetes.io/projected/630626e4-d3eb-4e14-8c9d-9363718e5049-kube-api-access-4vz24\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:15.005307 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.005153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1cd5923-8633-4b24-8310-e06ac0a2de68-metrics-tls\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.005307 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.005261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/630626e4-d3eb-4e14-8c9d-9363718e5049-cert\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:15.005450 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.005322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:15.005450 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.005357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cd5923-8633-4b24-8310-e06ac0a2de68-tmp-dir\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.098068 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.097881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:15.105918 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.105853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhg7\" (UniqueName: \"kubernetes.io/projected/b1cd5923-8633-4b24-8310-e06ac0a2de68-kube-api-access-nlhg7\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.106078 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.105954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz24\" (UniqueName: \"kubernetes.io/projected/630626e4-d3eb-4e14-8c9d-9363718e5049-kube-api-access-4vz24\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:15.106078 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.105988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1cd5923-8633-4b24-8310-e06ac0a2de68-metrics-tls\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.106078 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/630626e4-d3eb-4e14-8c9d-9363718e5049-cert\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:15.106078 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.106260 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnt9h\" (UniqueName: \"kubernetes.io/projected/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-api-access-dnt9h\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.106260 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cd5923-8633-4b24-8310-e06ac0a2de68-tmp-dir\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.106260 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/216e2c3d-4562-43b7-a06b-30743fa0cf0c-crio-socket\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.106391 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/216e2c3d-4562-43b7-a06b-30743fa0cf0c-data-volume\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.106391 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/216e2c3d-4562-43b7-a06b-30743fa0cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.106478 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cd5923-8633-4b24-8310-e06ac0a2de68-config-volume\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.106747 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.106708 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cd5923-8633-4b24-8310-e06ac0a2de68-tmp-dir\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.107186 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.107157 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cd5923-8633-4b24-8310-e06ac0a2de68-config-volume\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.114562 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.114495 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/630626e4-d3eb-4e14-8c9d-9363718e5049-cert\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:15.114832 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.114495 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1cd5923-8633-4b24-8310-e06ac0a2de68-metrics-tls\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.134182 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.134111 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhg7\" (UniqueName: \"kubernetes.io/projected/b1cd5923-8633-4b24-8310-e06ac0a2de68-kube-api-access-nlhg7\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.134443 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.134126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz24\" (UniqueName: \"kubernetes.io/projected/630626e4-d3eb-4e14-8c9d-9363718e5049-kube-api-access-4vz24\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:15.195099 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.195020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:15.203511 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.203466 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:15.207521 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.207448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/216e2c3d-4562-43b7-a06b-30743fa0cf0c-crio-socket\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.207910 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.207545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/216e2c3d-4562-43b7-a06b-30743fa0cf0c-data-volume\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.207910 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.207603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/216e2c3d-4562-43b7-a06b-30743fa0cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.207910 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.207703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.207910 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.207730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt9h\" (UniqueName: \"kubernetes.io/projected/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-api-access-dnt9h\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.207910 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.207781 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/216e2c3d-4562-43b7-a06b-30743fa0cf0c-crio-socket\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.208236 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.208184 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/216e2c3d-4562-43b7-a06b-30743fa0cf0c-data-volume\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.208647 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.208603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.211565 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.211494 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/216e2c3d-4562-43b7-a06b-30743fa0cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.232004 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.231904 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt9h\" (UniqueName: \"kubernetes.io/projected/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-api-access-dnt9h\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.301715 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.299929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:15.573907 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.573849 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9l55k"] Oct 30 18:01:15.577534 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.577493 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6mcgq"] Oct 30 18:01:15.580161 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:15.580111 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630626e4_d3eb_4e14_8c9d_9363718e5049.slice/crio-51c55f753b6c9d1b18977833eff0aad36398a4957953a5f84968a3e64eccef18 WatchSource:0}: Error finding container 51c55f753b6c9d1b18977833eff0aad36398a4957953a5f84968a3e64eccef18: Status 404 returned error can't find the container with id 51c55f753b6c9d1b18977833eff0aad36398a4957953a5f84968a3e64eccef18 Oct 30 18:01:15.581032 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:15.580993 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cd5923_8633_4b24_8310_e06ac0a2de68.slice/crio-d69e4d316b1946a705f8a7990d30cd7e8a19dcd97b6a095003e7217f9aa96b05 WatchSource:0}: Error finding container d69e4d316b1946a705f8a7990d30cd7e8a19dcd97b6a095003e7217f9aa96b05: Status 404 returned error can't find the container with id d69e4d316b1946a705f8a7990d30cd7e8a19dcd97b6a095003e7217f9aa96b05 Oct 30 18:01:15.644785 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.644612 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x6tdb"] Oct 30 18:01:15.650845 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:15.650786 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216e2c3d_4562_43b7_a06b_30743fa0cf0c.slice/crio-9075e2b792fc56fcf231b5d273c03e06d6c861f5aeee36057217024b25909971 WatchSource:0}: Error finding container 9075e2b792fc56fcf231b5d273c03e06d6c861f5aeee36057217024b25909971: Status 404 returned error can't find the container with id 9075e2b792fc56fcf231b5d273c03e06d6c861f5aeee36057217024b25909971 Oct 30 18:01:15.673586 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.673536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:15.673848 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.673536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:15.673848 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.673536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:15.722950 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.722873 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:01:15.723250 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.722899 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:01:15.723250 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.723043 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:01:15.723250 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.723131 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:01:15.723250 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.723225 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:01:15.727383 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.727335 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:01:15.727383 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:15.727338 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:01:16.035665 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.035396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:16.138549 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.138486 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:16.228594 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.228530 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9l55k" event={"ID":"630626e4-d3eb-4e14-8c9d-9363718e5049","Type":"ContainerStarted","Data":"51c55f753b6c9d1b18977833eff0aad36398a4957953a5f84968a3e64eccef18"} Oct 30 18:01:16.232022 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.231911 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6tdb" event={"ID":"216e2c3d-4562-43b7-a06b-30743fa0cf0c","Type":"ContainerStarted","Data":"da632e058be29ad1dc3613821413c47e3470442dfe4d2372a1c3439bfa07b1b1"} Oct 30 18:01:16.232374 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.232092 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6tdb" event={"ID":"216e2c3d-4562-43b7-a06b-30743fa0cf0c","Type":"ContainerStarted","Data":"9075e2b792fc56fcf231b5d273c03e06d6c861f5aeee36057217024b25909971"} Oct 30 18:01:16.235215 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.235132 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6mcgq" event={"ID":"b1cd5923-8633-4b24-8310-e06ac0a2de68","Type":"ContainerStarted","Data":"d69e4d316b1946a705f8a7990d30cd7e8a19dcd97b6a095003e7217f9aa96b05"} Oct 30 18:01:16.423029 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.422957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:16.576920 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.576843 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:16.677811 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:16.677661 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:17.247808 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:17.247729 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6tdb" event={"ID":"216e2c3d-4562-43b7-a06b-30743fa0cf0c","Type":"ContainerStarted","Data":"01382c6a34a15d6c83fe69a7e7d43cf95c437daf4c474dd30874cc31ef193f6f"} Oct 30 18:01:18.164070 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.164007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:18.257355 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.257291 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6mcgq" event={"ID":"b1cd5923-8633-4b24-8310-e06ac0a2de68","Type":"ContainerStarted","Data":"7a5c2e9ba7c2f93c8e87dd92d2c83e65b140d080870e3d6e09bafbedd6d25f70"} Oct 30 18:01:18.258112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.257441 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6mcgq" event={"ID":"b1cd5923-8633-4b24-8310-e06ac0a2de68","Type":"ContainerStarted","Data":"50fcf47c3db17c373f343d18a59b28e08dd7b49fb0ff82f20319cff128676cd7"} Oct 30 18:01:18.258112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.257569 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:18.327964 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.327857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:18.423115 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.421392 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-6mcgq" podStartSLOduration=2.405528579 podStartE2EDuration="4.421354035s" totalImagesPullingTime="2.015825456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:18.30412766 +0000 UTC m=+45.186866225" watchObservedRunningTime="2026-10-30 18:01:18.421354035 +0000 UTC m=+45.304092583" Oct 30 18:01:18.423115 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.422387 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29"] Oct 30 18:01:18.429101 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.429037 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.451108 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.451056 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29"] Oct 30 18:01:18.457570 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.456917 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 18:01:18.457570 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.457065 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:01:18.457570 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.457294 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:01:18.457570 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.457420 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hhk6k\"" Oct 30 18:01:18.457570 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.457496 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:01:18.462063 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.461995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:18.463350 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.463287 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:01:18.473840 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.473771 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7bjq6"] Oct 30 18:01:18.479088 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.479023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.484156 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.482167 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll"] Oct 30 18:01:18.487168 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.487113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.508838 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.508743 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll"] Oct 30 18:01:18.550137 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-sys\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.550482 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-metrics-client-ca\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.550482 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.550482 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-tls\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.550482 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.550698 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-wtmp\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.550698 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntf6\" (UniqueName: \"kubernetes.io/projected/3275088d-42df-41bd-8b42-b739a353962d-kube-api-access-vntf6\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.550698 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.550922 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2h77\" (UniqueName: \"kubernetes.io/projected/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-kube-api-access-z2h77\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.550922 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-textfile\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.551070 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.550926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.551070 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.551018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.551167 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.551116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-root\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.563099 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.563028 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:01:18.563398 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.563107 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 18:01:18.563398 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.563196 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 18:01:18.563398 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.563211 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:01:18.563398 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.563307 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h6qwn\"" Oct 30 18:01:18.563398 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.563371 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:01:18.563655 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.563408 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:01:18.563655 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.563493 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" Oct 30 18:01:18.573554 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.573487 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:18.652692 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.652619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-textfile\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.652692 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.652706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.652743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-wtmp\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.652774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.652820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-sys\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.652877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-metrics-client-ca\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.652910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.652997 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-sys\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.653159 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e2a6cc3b-391d-4626-908a-5466994e21d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.653554 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653175 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2h77\" (UniqueName: \"kubernetes.io/projected/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-kube-api-access-z2h77\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.653554 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bh65\" (UniqueName: \"kubernetes.io/projected/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-api-access-5bh65\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.653554 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-root\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.653554 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653325 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-root\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.653554 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653235 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-wtmp\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.654009 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.653968 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-textfile\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.654123 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.654123 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.654240 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-tls\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.654240 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.654341 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntf6\" (UniqueName: \"kubernetes.io/projected/3275088d-42df-41bd-8b42-b739a353962d-kube-api-access-vntf6\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.654341 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654327 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-metrics-client-ca\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.654450 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.654499 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.654671 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.654635 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.662614 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.662166 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.662614 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.662203 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.663049 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.662229 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.663552 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.663515 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-tls\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.678825 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.678673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:18.679135 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.679109 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntf6\" (UniqueName: \"kubernetes.io/projected/3275088d-42df-41bd-8b42-b739a353962d-kube-api-access-vntf6\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.681119 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.681079 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2h77\" (UniqueName: \"kubernetes.io/projected/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-kube-api-access-z2h77\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.755357 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.755274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.755357 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.755369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.755773 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.755636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.755773 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.755704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.755900 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.755777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e2a6cc3b-391d-4626-908a-5466994e21d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.755900 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.755835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh65\" (UniqueName: \"kubernetes.io/projected/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-api-access-5bh65\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.756682 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.756626 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.756972 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.756686 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.756972 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.756704 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e2a6cc3b-391d-4626-908a-5466994e21d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.760529 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.760473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.761001 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.760902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.765674 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.765580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:18.783768 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.783699 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh65\" (UniqueName: \"kubernetes.io/projected/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-api-access-5bh65\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.812957 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.812864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:18.834309 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.834082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:18.855610 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.855556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:18.871142 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:18.871068 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3275088d_42df_41bd_8b42_b739a353962d.slice/crio-76cd0ddb63e24298358d552efef0439c43d2191ce5d7da808491f211704edfcd WatchSource:0}: Error finding container 76cd0ddb63e24298358d552efef0439c43d2191ce5d7da808491f211704edfcd: Status 404 returned error can't find the container with id 76cd0ddb63e24298358d552efef0439c43d2191ce5d7da808491f211704edfcd Oct 30 18:01:18.986854 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:18.986669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:19.212847 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:19.212768 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29"] Oct 30 18:01:19.230923 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:19.230857 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll"] Oct 30 18:01:19.265980 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:19.265854 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x6tdb" event={"ID":"216e2c3d-4562-43b7-a06b-30743fa0cf0c","Type":"ContainerStarted","Data":"51853b4e72e1c8e3a199bced0e3f19d415c13fc7eb73e18b7bc5f63fa25bc101"} Oct 30 18:01:19.268701 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:19.268633 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bjq6" event={"ID":"3275088d-42df-41bd-8b42-b739a353962d","Type":"ContainerStarted","Data":"76cd0ddb63e24298358d552efef0439c43d2191ce5d7da808491f211704edfcd"} Oct 30 18:01:19.313574 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:19.313440 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x6tdb" podStartSLOduration=2.867412377 podStartE2EDuration="5.313406602s" totalImagesPullingTime="2.445994225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:01:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:19.312136168 +0000 UTC m=+46.194874723" watchObservedRunningTime="2026-10-30 18:01:19.313406602 +0000 UTC m=+46.196145156" Oct 30 18:01:19.395730 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:19.395579 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf49c75c_fb11_4c2d_ac07_b41d85f8f106.slice/crio-b2e2891508831f996f72384f06140e6016f74c014294ca16fd8d21e9eea7a8bb WatchSource:0}: Error finding container b2e2891508831f996f72384f06140e6016f74c014294ca16fd8d21e9eea7a8bb: Status 404 returned error can't find the container with id b2e2891508831f996f72384f06140e6016f74c014294ca16fd8d21e9eea7a8bb Oct 30 18:01:19.397554 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:19.397413 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a6cc3b_391d_4626_908a_5466994e21d5.slice/crio-065d6a58c6b07132554588819641138002ac6008325a7c236aeebc0d6802e3f8 WatchSource:0}: Error finding container 065d6a58c6b07132554588819641138002ac6008325a7c236aeebc0d6802e3f8: Status 404 returned error can't find the container with id 065d6a58c6b07132554588819641138002ac6008325a7c236aeebc0d6802e3f8 Oct 30 18:01:19.645055 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:19.644527 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:19.742801 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:19.742560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:19.903054 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:19.902973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:20.287539 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.287441 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9l55k" event={"ID":"630626e4-d3eb-4e14-8c9d-9363718e5049","Type":"ContainerStarted","Data":"07f034083f3a2764f9593b118ac6ecb9bf8ece8f62a140504920e64368320680"} Oct 30 18:01:20.292918 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.292822 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" event={"ID":"e2a6cc3b-391d-4626-908a-5466994e21d5","Type":"ContainerStarted","Data":"065d6a58c6b07132554588819641138002ac6008325a7c236aeebc0d6802e3f8"} Oct 30 18:01:20.298682 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.298604 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" event={"ID":"bf49c75c-fb11-4c2d-ac07-b41d85f8f106","Type":"ContainerStarted","Data":"40f4bd33f73ee5b7c39cfbf9322cd78701ec96ee6ef7fdce32528ffe2249a979"} Oct 30 18:01:20.299085 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.298744 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" event={"ID":"bf49c75c-fb11-4c2d-ac07-b41d85f8f106","Type":"ContainerStarted","Data":"336d648b60a8279dd6453781ed4f4ec64074d8a3696b6d073defa671e0d93969"} Oct 30 18:01:20.299085 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.298789 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" event={"ID":"bf49c75c-fb11-4c2d-ac07-b41d85f8f106","Type":"ContainerStarted","Data":"b2e2891508831f996f72384f06140e6016f74c014294ca16fd8d21e9eea7a8bb"} Oct 30 18:01:20.332673 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.332559 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9l55k" podStartSLOduration=2.489767881 podStartE2EDuration="6.332526933s" totalImagesPullingTime="3.842759052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:20.331604811 +0000 UTC m=+47.214343392" watchObservedRunningTime="2026-10-30 18:01:20.332526933 +0000 UTC m=+47.215265488" Oct 30 18:01:20.722314 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.722240 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74f988c6b9-jw4rr"] Oct 30 18:01:20.738794 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.738722 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.756674 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.754549 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74f988c6b9-jw4rr"] Oct 30 18:01:20.821825 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.821761 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 18:01:20.822164 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.821990 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 18:01:20.822164 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.821992 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 18:01:20.822164 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.822018 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 18:01:20.822164 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.822102 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 18:01:20.822305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.822176 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f9bu29dqg7ac4\"" Oct 30 18:01:20.822305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.822288 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8kj45\"" Oct 30 18:01:20.883354 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.883354 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.883742 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.883742 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-grpc-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.883742 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.883742 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.883742 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/269d1132-4f1d-4b95-b545-2b2659d245a7-tmp\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.883742 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxttb\" (UniqueName: \"kubernetes.io/projected/269d1132-4f1d-4b95-b545-2b2659d245a7-kube-api-access-gxttb\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.883742 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.883681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269d1132-4f1d-4b95-b545-2b2659d245a7-metrics-client-ca\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.985388 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.985229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/269d1132-4f1d-4b95-b545-2b2659d245a7-tmp\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.985388 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.985309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxttb\" (UniqueName: \"kubernetes.io/projected/269d1132-4f1d-4b95-b545-2b2659d245a7-kube-api-access-gxttb\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.985850 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.985806 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/269d1132-4f1d-4b95-b545-2b2659d245a7-tmp\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.985949 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.985882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269d1132-4f1d-4b95-b545-2b2659d245a7-metrics-client-ca\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.986123 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.986102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.986210 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.986140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.986210 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.986191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.986298 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.986236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-grpc-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.986572 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.986534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.986693 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.986610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.987055 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.987023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269d1132-4f1d-4b95-b545-2b2659d245a7-metrics-client-ca\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.994197 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.994140 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.994504 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.994140 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-grpc-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.994504 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.994140 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.994504 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.994350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.996319 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.996239 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:20.997085 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:20.997034 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:21.011778 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.011186 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxttb\" (UniqueName: \"kubernetes.io/projected/269d1132-4f1d-4b95-b545-2b2659d245a7-kube-api-access-gxttb\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:21.060341 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.060273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:21.143301 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.143228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:21.159253 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.159188 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:21.268985 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.268906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:21.316116 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.316056 2576 generic.go:356] "Generic (PLEG): container finished" podID="3275088d-42df-41bd-8b42-b739a353962d" containerID="0c50409e0912bbf0ec61d39aebe58622a155b2a0638f0d006830ab1cf7f30917" exitCode=0 Oct 30 18:01:21.317137 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.316119 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bjq6" event={"ID":"3275088d-42df-41bd-8b42-b739a353962d","Type":"ContainerDied","Data":"0c50409e0912bbf0ec61d39aebe58622a155b2a0638f0d006830ab1cf7f30917"} Oct 30 18:01:21.603366 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.602619 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74f988c6b9-jw4rr"] Oct 30 18:01:21.620120 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:21.620061 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269d1132_4f1d_4b95_b545_2b2659d245a7.slice/crio-b69283a81066adf3eba34d6a9de0e1812fc164e2ed5c4905fef813c09c763af2 WatchSource:0}: Error finding container b69283a81066adf3eba34d6a9de0e1812fc164e2ed5c4905fef813c09c763af2: Status 404 returned error can't find the container with id b69283a81066adf3eba34d6a9de0e1812fc164e2ed5c4905fef813c09c763af2 Oct 30 18:01:21.779407 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.778837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:21.964668 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:21.964615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:22.330072 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.329996 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bjq6" event={"ID":"3275088d-42df-41bd-8b42-b739a353962d","Type":"ContainerStarted","Data":"97967957aff12d586f70ff97a9588dda6e87acdbbbef512418dac5391b0c077e"} Oct 30 18:01:22.331577 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.330106 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7bjq6" event={"ID":"3275088d-42df-41bd-8b42-b739a353962d","Type":"ContainerStarted","Data":"355b69b1c3f4d7e0bc42ceb785f3d05c4d503771ff9daa043f9be20d0696924c"} Oct 30 18:01:22.335461 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.335391 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" event={"ID":"bf49c75c-fb11-4c2d-ac07-b41d85f8f106","Type":"ContainerStarted","Data":"130b68b215e5e0cd769ccbdc8286513d7d2d1278be74b9be6e123bcb8cb090dc"} Oct 30 18:01:22.338247 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.338176 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" event={"ID":"269d1132-4f1d-4b95-b545-2b2659d245a7","Type":"ContainerStarted","Data":"b69283a81066adf3eba34d6a9de0e1812fc164e2ed5c4905fef813c09c763af2"} Oct 30 18:01:22.343902 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.343808 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" event={"ID":"e2a6cc3b-391d-4626-908a-5466994e21d5","Type":"ContainerStarted","Data":"901d932057c932522a190a921f8265dc2db7e953e9ecba476968f03659df8c33"} Oct 30 18:01:22.344510 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.343984 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" event={"ID":"e2a6cc3b-391d-4626-908a-5466994e21d5","Type":"ContainerStarted","Data":"6cd08f8f12d01c6018373264f5598014c4a4700e74edf3650d2ea097bf9df0b4"} Oct 30 18:01:22.344510 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.344028 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" event={"ID":"e2a6cc3b-391d-4626-908a-5466994e21d5","Type":"ContainerStarted","Data":"fd9dfe57d79f9632e52117806f1d3365a00ed5cfa949d341446c87daf1e6d590"} Oct 30 18:01:22.380390 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.380261 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7bjq6" podStartSLOduration=2.772217235 podStartE2EDuration="4.380224247s" totalImagesPullingTime="1.451314734s" totalInitContainerRuntime="156.692278ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:22.37819221 +0000 UTC m=+49.260930766" watchObservedRunningTime="2026-10-30 18:01:22.380224247 +0000 UTC m=+49.262962802" Oct 30 18:01:22.424623 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.424512 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" podStartSLOduration=2.890940691 podStartE2EDuration="4.424485565s" totalImagesPullingTime="1.533544874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:22.424218619 +0000 UTC m=+49.306957193" watchObservedRunningTime="2026-10-30 18:01:22.424485565 +0000 UTC m=+49.307224117" Oct 30 18:01:22.472797 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.472644 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" podStartSLOduration=2.456948963 podStartE2EDuration="4.472604999s" totalImagesPullingTime="2.015656036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:22.47072919 +0000 UTC m=+49.353467736" watchObservedRunningTime="2026-10-30 18:01:22.472604999 +0000 UTC m=+49.355343550" Oct 30 18:01:22.945633 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:22.945525 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:23.049217 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.049140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:23.276490 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.276321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:23.301576 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.301498 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f87jc"] Oct 30 18:01:23.320566 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.320505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:23.325706 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.325629 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f87jc"] Oct 30 18:01:23.337531 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.337443 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 18:01:23.340611 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.340550 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q92vx\"" Oct 30 18:01:23.420780 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.420559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e62ca62-c183-4859-8d52-a52aa8182c73-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:23.420780 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.420724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8nm\" (UniqueName: \"kubernetes.io/projected/9e62ca62-c183-4859-8d52-a52aa8182c73-kube-api-access-ch8nm\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:23.514441 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.514366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:23.521801 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.521698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e62ca62-c183-4859-8d52-a52aa8182c73-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:23.522414 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.522363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8nm\" (UniqueName: \"kubernetes.io/projected/9e62ca62-c183-4859-8d52-a52aa8182c73-kube-api-access-ch8nm\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:23.529305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.529179 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e62ca62-c183-4859-8d52-a52aa8182c73-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:23.548461 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.548381 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8nm\" (UniqueName: \"kubernetes.io/projected/9e62ca62-c183-4859-8d52-a52aa8182c73-kube-api-access-ch8nm\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:23.619136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.619081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:23.656720 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:23.656654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:24.021692 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:24.021559 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-f87jc"] Oct 30 18:01:24.202227 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:24.202174 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e62ca62_c183_4859_8d52_a52aa8182c73.slice/crio-4bd29c553ab4bdfb87cf14071818c01fe1b7ce0c729a3e3fc690309609850121 WatchSource:0}: Error finding container 4bd29c553ab4bdfb87cf14071818c01fe1b7ce0c729a3e3fc690309609850121: Status 404 returned error can't find the container with id 4bd29c553ab4bdfb87cf14071818c01fe1b7ce0c729a3e3fc690309609850121 Oct 30 18:01:24.361045 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:24.360976 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" event={"ID":"9e62ca62-c183-4859-8d52-a52aa8182c73","Type":"ContainerStarted","Data":"4bd29c553ab4bdfb87cf14071818c01fe1b7ce0c729a3e3fc690309609850121"} Oct 30 18:01:25.374060 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.373970 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" event={"ID":"269d1132-4f1d-4b95-b545-2b2659d245a7","Type":"ContainerStarted","Data":"408c1c59389d03ae3b4daf2a00210348ac636aec5cb27a087ba456fed19c77ac"} Oct 30 18:01:25.374905 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.374207 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" event={"ID":"269d1132-4f1d-4b95-b545-2b2659d245a7","Type":"ContainerStarted","Data":"fb6d45c5988a6dee5510a2bd966bd5b73cff713eb44110eb5d6921e360998049"} Oct 30 18:01:25.374905 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.374287 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" event={"ID":"269d1132-4f1d-4b95-b545-2b2659d245a7","Type":"ContainerStarted","Data":"a7a309953675a14805e968314211b4289985011f54ecbb14f43853036af0a161"} Oct 30 18:01:25.482118 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.482045 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:01:25.501120 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.501065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.513870 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.513800 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:01:25.520150 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.520079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:25.525336 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.525276 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:01:25.525703 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.525479 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:01:25.527742 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.527686 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:01:25.539894 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.539821 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:01:25.539894 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.539820 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:01:25.545719 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.545719 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.545997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546028 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546112 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546832 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546832 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546832 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546832 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbjpt\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546832 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.546832 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.546263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.553831 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.553831 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.553891 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.553950 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.553975 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.553984 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.553998 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.554030 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.554035 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:01:25.554136 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.554103 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:01:25.554724 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.554236 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gzppf\"" Oct 30 18:01:25.554724 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.554330 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18mfosc96e6jj\"" Oct 30 18:01:25.647879 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.647712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.647879 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.647813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648258 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.647902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648258 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648003 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648258 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648258 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648258 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648258 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648258 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648597 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648597 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648688 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648740 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648793 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648839 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.648908 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.649000 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.649000 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.648986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.649111 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.649030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.649111 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.649074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjpt\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.649224 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.649191 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.649283 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.649240 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.649333 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.649222 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.649919 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.649617 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.652276 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.652216 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.653336 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.653058 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.653621 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.653449 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.654640 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.654562 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.654928 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.654679 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.656016 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.655246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.656016 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.655891 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.656292 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.656119 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.657132 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.657046 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.659195 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.659137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.659522 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.659439 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.659814 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.659781 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.664305 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.664246 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.664621 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.664593 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.669066 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.669014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.673738 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.673664 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjpt\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:25.721680 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:25.721601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:26.384645 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:26.384576 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" event={"ID":"269d1132-4f1d-4b95-b545-2b2659d245a7","Type":"ContainerStarted","Data":"fabf2034e799081a11df2118889c7c96a3687039b203809ba9422d63fd3af00f"} Oct 30 18:01:26.387879 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:26.387802 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" event={"ID":"9e62ca62-c183-4859-8d52-a52aa8182c73","Type":"ContainerStarted","Data":"fc2fba0c2ca4a34704a972466742e928b360717abcbcbbdf0a67c100aec215ef"} Oct 30 18:01:26.388255 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:26.388197 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:26.396434 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:26.396374 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:26.432258 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:26.432160 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" podStartSLOduration=1.489825526 podStartE2EDuration="3.432126471s" totalImagesPullingTime="1.942300945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:26.431345501 +0000 UTC m=+53.314084073" watchObservedRunningTime="2026-10-30 18:01:26.432126471 +0000 UTC m=+53.314865024" Oct 30 18:01:26.519573 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:26.519512 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:01:26.526578 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:26.526509 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1afeda_d283_47f1_978a_43aeeae7d6cb.slice/crio-c627932e68de4b280509b3b4a67ba6ec8dba4dc843e0bfb4d1421cc4b8c8124a WatchSource:0}: Error finding container c627932e68de4b280509b3b4a67ba6ec8dba4dc843e0bfb4d1421cc4b8c8124a: Status 404 returned error can't find the container with id c627932e68de4b280509b3b4a67ba6ec8dba4dc843e0bfb4d1421cc4b8c8124a Oct 30 18:01:27.192710 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:27.192639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:27.333637 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:27.333558 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:27.402273 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:27.402193 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" event={"ID":"269d1132-4f1d-4b95-b545-2b2659d245a7","Type":"ContainerStarted","Data":"79bb91b5db631bbbb044f7422230d13f38118b475249b06d409b4f47e21af899"} Oct 30 18:01:27.403059 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:27.402360 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" event={"ID":"269d1132-4f1d-4b95-b545-2b2659d245a7","Type":"ContainerStarted","Data":"838ed612bd1810a50414d291316fc0c016b854d6a4e6104ccd8e375e6cdfd453"} Oct 30 18:01:27.403059 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:27.402435 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:27.405921 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:27.405848 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerStarted","Data":"c627932e68de4b280509b3b4a67ba6ec8dba4dc843e0bfb4d1421cc4b8c8124a"} Oct 30 18:01:27.481745 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:27.481544 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" podStartSLOduration=3.325485862 podStartE2EDuration="7.481506658s" totalImagesPullingTime="4.156020796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:27.466449393 +0000 UTC m=+54.349187972" watchObservedRunningTime="2026-10-30 18:01:27.481506658 +0000 UTC m=+54.364245215" Oct 30 18:01:28.165295 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:28.165212 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:28.269858 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:28.269802 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:28.272535 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:28.272476 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:28.376561 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:28.376502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:28.419353 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:28.419201 2576 generic.go:356] "Generic (PLEG): container finished" podID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerID="eeaa04d40d7f51db91196146c7868ea79da05945fb9a4f55784220ab9404ea8c" exitCode=0 Oct 30 18:01:28.419353 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:28.419298 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerDied","Data":"eeaa04d40d7f51db91196146c7868ea79da05945fb9a4f55784220ab9404ea8c"} Oct 30 18:01:30.682540 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:30.682429 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:30.779482 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:30.779421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:32.196836 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.196788 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:32.450146 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.449996 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerStarted","Data":"2e9a900703b978498f1d64f21da42dee80524b69c8cd216558c8c14816485d12"} Oct 30 18:01:32.450146 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.450115 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerStarted","Data":"b034530536b45df40c271a308eb4c45a2178b363d6965baf087b534fdb50054a"} Oct 30 18:01:32.450459 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.450157 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerStarted","Data":"d3eeeea5025d5952b94ab67d83d3ad002c7d433fe276419aa3bf7e2cae46658c"} Oct 30 18:01:32.450459 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.450193 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerStarted","Data":"970d5eb05250e54cd9fad61f9bc850db3b2dfc29300b17cbe9fb5fd2bd924e36"} Oct 30 18:01:32.450459 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.450230 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerStarted","Data":"67879e3838023e126dd9d1dcc52fdfc864bcbef6d23dcf11c8e9b35480ee4120"} Oct 30 18:01:32.450459 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.450265 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerStarted","Data":"de0a1c6b124af964f263dcb8709d0187ff15a67ef8ebcd2fa89a2d371318f8cd"} Oct 30 18:01:32.506165 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.506069 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.135970954 podStartE2EDuration="7.506034223s" totalImagesPullingTime="4.342233369s" totalInitContainerRuntime="27.8299ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:32.504669094 +0000 UTC m=+59.387407649" watchObservedRunningTime="2026-10-30 18:01:32.506034223 +0000 UTC m=+59.388772780" Oct 30 18:01:32.734085 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.733896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:32.823217 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.823104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:32.909102 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:32.909047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:33.042196 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:33.042064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:33.430104 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:33.430058 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:35.592042 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:35.591954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:35.692208 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:35.692119 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:35.722420 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:35.722348 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:37.869656 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:37.869550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:37.969255 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:37.969187 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:38.472549 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:38.472474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:38.592566 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:38.592512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:39.431502 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.431417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:39.447395 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.447351 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:01:39.456384 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.456334 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:39.532799 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.532729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:39.532799 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.532815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:39.548603 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.548553 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:01:39.548809 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.548713 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:01:39.557793 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.557715 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:39.569271 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.569215 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:01:39.585111 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.585043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:39.731039 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.730872 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:01:39.735424 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.735356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:39.756594 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.756548 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:01:39.761577 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.761504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:39.768558 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.768509 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:01:39.773485 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:39.773416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:40.079740 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:40.079678 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qg2lp"] Oct 30 18:01:40.084024 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:40.083833 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9cf883_7d59_46f0_b457_f84c7c678251.slice/crio-a0fa182c61696ac9605b93fd674d0ad614bb1310372893f63ce1f4bb4b020271 WatchSource:0}: Error finding container a0fa182c61696ac9605b93fd674d0ad614bb1310372893f63ce1f4bb4b020271: Status 404 returned error can't find the container with id a0fa182c61696ac9605b93fd674d0ad614bb1310372893f63ce1f4bb4b020271 Oct 30 18:01:40.111104 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:40.111053 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpbwl"] Oct 30 18:01:40.114409 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:40.114348 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4312f3_14a2_49b4_a6f1_4484d94a2217.slice/crio-4f88e2dc7b92290621e92caf53d9bad2eef53ef513ba6b898462bc3a83fbbbc6 WatchSource:0}: Error finding container 4f88e2dc7b92290621e92caf53d9bad2eef53ef513ba6b898462bc3a83fbbbc6: Status 404 returned error can't find the container with id 4f88e2dc7b92290621e92caf53d9bad2eef53ef513ba6b898462bc3a83fbbbc6 Oct 30 18:01:40.142326 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:40.142237 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hchmm"] Oct 30 18:01:40.145187 ip-10-0-133-47 kubenswrapper[2576]: W1030 18:01:40.145123 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbee04d_3f06_4ce5_8020_4d52b9715887.slice/crio-cab59a04bf4c6c1297a19aecc6b8646be0695e1afe452bea91001a6c0de18f5e WatchSource:0}: Error finding container cab59a04bf4c6c1297a19aecc6b8646be0695e1afe452bea91001a6c0de18f5e: Status 404 returned error can't find the container with id cab59a04bf4c6c1297a19aecc6b8646be0695e1afe452bea91001a6c0de18f5e Oct 30 18:01:40.498064 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:40.497983 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpbwl" event={"ID":"7e4312f3-14a2-49b4-a6f1-4484d94a2217","Type":"ContainerStarted","Data":"4f88e2dc7b92290621e92caf53d9bad2eef53ef513ba6b898462bc3a83fbbbc6"} Oct 30 18:01:40.500869 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:40.500795 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hchmm" event={"ID":"0bbee04d-3f06-4ce5-8020-4d52b9715887","Type":"ContainerStarted","Data":"cab59a04bf4c6c1297a19aecc6b8646be0695e1afe452bea91001a6c0de18f5e"} Oct 30 18:01:40.503663 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:40.503383 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qg2lp" event={"ID":"8e9cf883-7d59-46f0-b457-f84c7c678251","Type":"ContainerStarted","Data":"a0fa182c61696ac9605b93fd674d0ad614bb1310372893f63ce1f4bb4b020271"} Oct 30 18:01:41.383793 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:41.383740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:41.484902 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:41.484837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:41.585640 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:41.585573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:42.521486 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:42.521386 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hchmm" event={"ID":"0bbee04d-3f06-4ce5-8020-4d52b9715887","Type":"ContainerStarted","Data":"a9563cb41df8d239629730b681783359d938b1034aeeb663a7d219b3ff7db78a"} Oct 30 18:01:42.521858 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:42.521556 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hchmm" event={"ID":"0bbee04d-3f06-4ce5-8020-4d52b9715887","Type":"ContainerStarted","Data":"33c1507dd037cfd02b951b9cdf9428c06cc3af91e9e3b15c1a11a0f35ecf3f36"} Oct 30 18:01:42.574076 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:42.566718 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hchmm" podStartSLOduration=67.329904462 podStartE2EDuration="1m8.566674545s" totalImagesPullingTime="1.236770083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:42.564430358 +0000 UTC m=+69.447168912" watchObservedRunningTime="2026-10-30 18:01:42.566674545 +0000 UTC m=+69.449413107" Oct 30 18:01:44.540801 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:44.540713 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpbwl" event={"ID":"7e4312f3-14a2-49b4-a6f1-4484d94a2217","Type":"ContainerStarted","Data":"05b878940072c48e66de60455e66112b1acf41611e0d2e9d4b992e0e68159943"} Oct 30 18:01:44.541663 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:44.541105 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:44.583833 ip-10-0-133-47 kubenswrapper[2576]: I1030 18:01:44.583706 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xpbwl" podStartSLOduration=66.755935186 podStartE2EDuration="1m10.58367272s" totalImagesPullingTime="3.827737534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:44.583486106 +0000 UTC m=+71.466224661" watchObservedRunningTime="2026-10-30 18:01:44.58367272 +0000 UTC m=+71.466411275" Oct 30 18:01:45.350780 ip-10-0-133-47 kubenswrapper[2576]: E1030 18:01:45.350707 2576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:01:45.502142 ip-10-0-133-47 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 18:01:45.524745 ip-10-0-133-47 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 18:01:45.525166 ip-10-0-133-47 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 18:01:45.525591 ip-10-0-133-47 systemd[1]: kubelet.service: Consumed 8.204s CPU time. Oct 30 18:01:45.543454 ip-10-0-133-47 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:01:45.658897 ip-10-0-133-47 kubenswrapper[5624]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:01:45.658897 ip-10-0-133-47 kubenswrapper[5624]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:01:45.658897 ip-10-0-133-47 kubenswrapper[5624]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:01:45.658897 ip-10-0-133-47 kubenswrapper[5624]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:01:45.661878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661641 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:45.661878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661866 5624 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:45.661878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661870 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:45.661878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661878 5624 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:45.661878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661885 5624 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:45.661878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661890 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:45.661878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661893 5624 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:45.661878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661897 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661902 5624 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661905 5624 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661909 5624 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661912 5624 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661916 5624 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661918 5624 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661921 5624 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661923 5624 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661926 5624 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661952 5624 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661956 5624 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661959 5624 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661962 5624 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661966 5624 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661969 5624 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661973 5624 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661976 5624 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661980 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:45.662171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661983 5624 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.661995 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662000 5624 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662003 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662006 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662009 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662012 5624 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662015 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662018 5624 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662021 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662026 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662029 5624 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662033 5624 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662036 5624 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662040 5624 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662044 5624 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662047 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662052 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662056 5624 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:45.662756 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662060 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662064 5624 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662067 5624 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662071 5624 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662074 5624 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662077 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662083 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662087 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662090 5624 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662093 5624 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662097 5624 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662099 5624 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662102 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662105 5624 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662108 5624 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662112 5624 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662114 5624 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662117 5624 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662120 5624 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662122 5624 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:45.663395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662125 5624 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662128 5624 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662131 5624 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662134 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662137 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662139 5624 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662142 5624 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662144 5624 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662147 5624 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662150 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662153 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662156 5624 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662159 5624 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662163 5624 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662167 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662171 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662175 5624 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662178 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662181 5624 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:45.664038 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662185 5624 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662188 5624 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662190 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662193 5624 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662196 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662199 5624 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662205 5624 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662209 5624 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662213 5624 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662218 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662222 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662225 5624 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662229 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662231 5624 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662235 5624 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662238 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662240 5624 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662244 5624 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662247 5624 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662250 5624 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:45.664571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662258 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.662261 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663483 5624 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663501 5624 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663504 5624 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663508 5624 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663511 5624 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663514 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663518 5624 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663522 5624 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663526 5624 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663529 5624 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663532 5624 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663535 5624 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663539 5624 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663542 5624 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663546 5624 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663550 5624 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663553 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663556 5624 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:45.665246 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663559 5624 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663562 5624 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663576 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663588 5624 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663592 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663596 5624 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663599 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663602 5624 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663605 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663608 5624 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663611 5624 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663614 5624 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663617 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663620 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663623 5624 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663628 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663631 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663636 5624 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663642 5624 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:45.665801 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663646 5624 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663649 5624 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663651 5624 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663654 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663657 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663660 5624 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663662 5624 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663665 5624 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663667 5624 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663670 5624 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663673 5624 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663676 5624 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663679 5624 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663681 5624 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663684 5624 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663687 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663702 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663705 5624 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663708 5624 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663711 5624 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:45.666421 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663714 5624 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663717 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663720 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663723 5624 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663726 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663729 5624 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663732 5624 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663734 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663738 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663741 5624 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663745 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663748 5624 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663750 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663753 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663756 5624 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663759 5624 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663762 5624 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663765 5624 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663768 5624 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:45.667025 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663773 5624 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663777 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663821 5624 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663826 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663829 5624 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663834 5624 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663838 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663841 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663847 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663853 5624 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663858 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663863 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663876 5624 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663880 5624 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663884 5624 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663889 5624 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663892 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663896 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663899 5624 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:45.667549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663903 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663907 5624 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663910 5624 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663914 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663916 5624 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663920 5624 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663924 5624 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663927 5624 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663953 5624 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663957 5624 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.663963 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664723 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664734 5624 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664739 5624 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664742 5624 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664746 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664749 5624 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664752 5624 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664756 5624 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:45.668170 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664759 5624 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664762 5624 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664766 5624 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664770 5624 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664774 5624 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664777 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664780 5624 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664783 5624 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664786 5624 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664790 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664793 5624 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664796 5624 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664799 5624 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664802 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664805 5624 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664808 5624 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664812 5624 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664814 5624 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664817 5624 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664820 5624 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:45.668698 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664823 5624 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664826 5624 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664829 5624 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664832 5624 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664835 5624 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664838 5624 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664841 5624 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664844 5624 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664846 5624 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664850 5624 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664853 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664855 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664858 5624 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664861 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664863 5624 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664867 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664870 5624 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664874 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664877 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:45.669305 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664882 5624 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664887 5624 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664890 5624 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664893 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664896 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664898 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664902 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664913 5624 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664916 5624 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664919 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664922 5624 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664924 5624 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664927 5624 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664951 5624 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664954 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664957 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664960 5624 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664963 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664968 5624 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:45.669849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664973 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664977 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664981 5624 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664984 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664987 5624 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664990 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664993 5624 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.664997 5624 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665000 5624 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665003 5624 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665006 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665010 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665014 5624 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665018 5624 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665021 5624 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665024 5624 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665027 5624 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665031 5624 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665033 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:45.670703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665036 5624 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665043 5624 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665046 5624 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665049 5624 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665052 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665055 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665057 5624 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665061 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665063 5624 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665067 5624 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665070 5624 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665072 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665075 5624 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665079 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665081 5624 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665085 5624 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665088 5624 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665091 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665093 5624 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665096 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:45.671345 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.665099 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665265 5624 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665279 5624 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665288 5624 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665296 5624 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665303 5624 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665311 5624 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665322 5624 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665328 5624 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665333 5624 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665337 5624 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665342 5624 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665346 5624 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665350 5624 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665354 5624 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665358 5624 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665361 5624 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665366 5624 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665369 5624 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665375 5624 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665378 5624 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665381 5624 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665385 5624 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665390 5624 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:01:45.671929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665396 5624 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665400 5624 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665407 5624 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665412 5624 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665416 5624 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665420 5624 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665425 5624 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665428 5624 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665432 5624 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665443 5624 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665447 5624 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665450 5624 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665454 5624 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665457 5624 flags.go:64] FLAG: --enable-server="true" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665461 5624 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665469 5624 flags.go:64] FLAG: --event-burst="100" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665473 5624 flags.go:64] FLAG: --event-qps="50" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665477 5624 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665481 5624 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665486 5624 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665492 5624 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665496 5624 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665499 5624 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665503 5624 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665507 5624 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:01:45.672636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665513 5624 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665517 5624 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665520 5624 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665523 5624 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665527 5624 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665531 5624 flags.go:64] FLAG: --feature-gates="" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665538 5624 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665542 5624 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665546 5624 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665550 5624 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665554 5624 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665558 5624 flags.go:64] FLAG: --help="false" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665562 5624 flags.go:64] FLAG: --hostname-override="ip-10-0-133-47.ec2.internal" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665567 5624 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665570 5624 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665576 5624 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665585 5624 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665589 5624 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665592 5624 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665597 5624 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665602 5624 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665606 5624 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665610 5624 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665616 5624 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:01:45.673476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665619 5624 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665622 5624 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665626 5624 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665630 5624 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665633 5624 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665637 5624 flags.go:64] FLAG: --lock-file="" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665641 5624 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665645 5624 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665648 5624 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665658 5624 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665661 5624 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665665 5624 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665669 5624 flags.go:64] FLAG: --logging-format="text" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665672 5624 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665677 5624 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665681 5624 flags.go:64] FLAG: --manifest-url="" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665684 5624 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665693 5624 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665698 5624 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665705 5624 flags.go:64] FLAG: --max-pods="110" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665709 5624 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665713 5624 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665716 5624 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665720 5624 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665728 5624 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:01:45.674394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665732 5624 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665735 5624 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665748 5624 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665752 5624 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665757 5624 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665761 5624 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665767 5624 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665772 5624 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665779 5624 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665783 5624 flags.go:64] FLAG: --port="10250" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665787 5624 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665791 5624 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f35d4c477ea95aee" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665794 5624 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665798 5624 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665802 5624 flags.go:64] FLAG: --register-node="true" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665806 5624 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665812 5624 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665817 5624 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665824 5624 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665828 5624 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665834 5624 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665838 5624 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665842 5624 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665848 5624 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665852 5624 flags.go:64] FLAG: --runonce="false" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665856 5624 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:01:45.675472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665860 5624 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665864 5624 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665868 5624 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665871 5624 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665875 5624 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665883 5624 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665888 5624 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665892 5624 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665896 5624 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665899 5624 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665903 5624 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665908 5624 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665912 5624 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665916 5624 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665928 5624 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665954 5624 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665961 5624 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665969 5624 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665976 5624 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665981 5624 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665985 5624 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665988 5624 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.665993 5624 flags.go:64] FLAG: --v="2" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.666000 5624 flags.go:64] FLAG: --version="false" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.666015 5624 flags.go:64] FLAG: --vmodule="" Oct 30 18:01:45.676560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.666022 5624 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.666026 5624 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666256 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666263 5624 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666267 5624 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666273 5624 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666276 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666279 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666282 5624 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666286 5624 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666289 5624 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666293 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666302 5624 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666305 5624 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666308 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666311 5624 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666315 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666318 5624 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666322 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:45.677578 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666325 5624 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666328 5624 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666332 5624 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666337 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666340 5624 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666346 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666349 5624 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666351 5624 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666354 5624 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666357 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666361 5624 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666364 5624 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666371 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666373 5624 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666376 5624 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666380 5624 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666385 5624 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666388 5624 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666391 5624 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666394 5624 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:45.678150 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666397 5624 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666400 5624 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666403 5624 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666405 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666408 5624 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666415 5624 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666418 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666422 5624 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666426 5624 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666429 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666433 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666436 5624 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666439 5624 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666442 5624 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666444 5624 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666447 5624 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666452 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666454 5624 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666457 5624 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666460 5624 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:45.678703 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666463 5624 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666467 5624 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666469 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666473 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666479 5624 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666482 5624 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666484 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666488 5624 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666491 5624 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666494 5624 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666497 5624 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666499 5624 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666502 5624 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666505 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666509 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666512 5624 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666515 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666521 5624 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666526 5624 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:45.679308 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666530 5624 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666534 5624 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666537 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666539 5624 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666544 5624 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666549 5624 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666552 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666556 5624 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666559 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666562 5624 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666565 5624 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666571 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666574 5624 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666577 5624 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666580 5624 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666583 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666585 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666593 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666596 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:45.679905 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666599 5624 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666603 5624 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666605 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666608 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666611 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666613 5624 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666617 5624 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666620 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666623 5624 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666627 5624 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666630 5624 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.666684 5624 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666741 5624 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666748 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666752 5624 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666757 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:45.680610 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666761 5624 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666764 5624 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666768 5624 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666771 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666774 5624 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666778 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666782 5624 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666784 5624 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666788 5624 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666791 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666795 5624 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666797 5624 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666800 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666803 5624 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666809 5624 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666812 5624 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666815 5624 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666818 5624 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666821 5624 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666824 5624 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:45.681139 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666827 5624 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666830 5624 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666833 5624 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666835 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666838 5624 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666841 5624 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666845 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666847 5624 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666853 5624 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666856 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666858 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666862 5624 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666865 5624 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666868 5624 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666870 5624 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666874 5624 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666877 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666880 5624 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666883 5624 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666886 5624 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:45.681693 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666890 5624 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666894 5624 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666897 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666900 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666904 5624 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666907 5624 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666914 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666918 5624 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666921 5624 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666924 5624 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666927 5624 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666952 5624 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666957 5624 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666959 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666962 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666965 5624 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666968 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666973 5624 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666977 5624 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:45.682275 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666980 5624 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666988 5624 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666991 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.666995 5624 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667000 5624 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667003 5624 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667007 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667010 5624 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667013 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667016 5624 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667020 5624 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667025 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667029 5624 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667032 5624 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667036 5624 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667041 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667044 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667048 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667051 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:45.682903 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667057 5624 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667061 5624 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667063 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667067 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667071 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667074 5624 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667077 5624 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667080 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667083 5624 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667086 5624 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667089 5624 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667092 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667095 5624 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667098 5624 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667103 5624 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667106 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667109 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667111 5624 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667114 5624 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:45.683514 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667116 5624 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667120 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667124 5624 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667127 5624 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.667129 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.674894 5624 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.674982 5624 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.675023 5624 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.675033 5624 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675287 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675300 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675311 5624 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675320 5624 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675326 5624 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675331 5624 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675336 5624 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675341 5624 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675345 5624 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675350 5624 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:45.684081 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675362 5624 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675367 5624 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675372 5624 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675378 5624 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675384 5624 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675389 5624 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675394 5624 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675399 5624 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675404 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675410 5624 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675415 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675419 5624 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675424 5624 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675436 5624 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675442 5624 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675447 5624 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675452 5624 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675458 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675464 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675474 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:45.684631 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675480 5624 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675485 5624 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675492 5624 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675498 5624 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675505 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675516 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675521 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675526 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675531 5624 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675536 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675545 5624 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675551 5624 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675556 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675561 5624 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675565 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675569 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675574 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675579 5624 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675656 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675722 5624 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:45.685276 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675732 5624 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675742 5624 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675752 5624 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675759 5624 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675765 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675771 5624 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675779 5624 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675786 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675792 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675806 5624 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675812 5624 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675860 5624 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.675866 5624 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676124 5624 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676148 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676157 5624 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676162 5624 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676167 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676170 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:45.685873 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676174 5624 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676177 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676182 5624 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676187 5624 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676192 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676197 5624 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676204 5624 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676210 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676214 5624 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676218 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676222 5624 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676225 5624 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676232 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676235 5624 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676241 5624 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676245 5624 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676248 5624 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676252 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676255 5624 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676259 5624 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:45.686739 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676263 5624 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676267 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676270 5624 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676274 5624 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676279 5624 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676284 5624 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676288 5624 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676293 5624 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676297 5624 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676302 5624 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676307 5624 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676310 5624 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676313 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676316 5624 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676320 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676328 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:45.687443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676333 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.676465 5624 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676686 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676699 5624 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676703 5624 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676707 5624 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676711 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676714 5624 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676719 5624 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676722 5624 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676726 5624 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676729 5624 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676733 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676736 5624 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676739 5624 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676742 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:45.687892 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676745 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676748 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676751 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676754 5624 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676757 5624 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676760 5624 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676763 5624 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676766 5624 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676768 5624 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676772 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676775 5624 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676779 5624 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676782 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676785 5624 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676788 5624 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676793 5624 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676796 5624 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676800 5624 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676803 5624 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676806 5624 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:45.688378 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676809 5624 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676812 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676815 5624 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676817 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676824 5624 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676828 5624 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676832 5624 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676836 5624 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676840 5624 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676843 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676846 5624 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676849 5624 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676852 5624 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676855 5624 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676859 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676862 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676866 5624 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676870 5624 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676873 5624 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:45.689015 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676879 5624 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676884 5624 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676889 5624 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676894 5624 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676899 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676903 5624 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676906 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676910 5624 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676915 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676919 5624 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676925 5624 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676928 5624 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676958 5624 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676964 5624 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676968 5624 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676973 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676979 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676984 5624 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676990 5624 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.676997 5624 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:45.689583 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677002 5624 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677006 5624 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677010 5624 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677015 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677022 5624 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677027 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677032 5624 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677037 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677042 5624 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677047 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677052 5624 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677058 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677063 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677070 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677076 5624 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677083 5624 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677090 5624 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677096 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677102 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:45.690360 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677106 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677110 5624 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677114 5624 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677118 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677122 5624 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677125 5624 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677128 5624 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677131 5624 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677136 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677139 5624 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677141 5624 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677145 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677150 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677154 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.677224 5624 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677275 5624 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:45.691096 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677282 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677287 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677292 5624 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677295 5624 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677298 5624 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677302 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677304 5624 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677308 5624 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677312 5624 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677314 5624 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677318 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677323 5624 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677326 5624 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677342 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677346 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677350 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677353 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677357 5624 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677360 5624 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677363 5624 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:45.691667 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677365 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677369 5624 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677372 5624 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677375 5624 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677378 5624 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677382 5624 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677385 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677388 5624 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677391 5624 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677396 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677399 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677403 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677406 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677408 5624 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677411 5624 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677414 5624 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677417 5624 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677420 5624 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677424 5624 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677427 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:45.692549 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677430 5624 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677433 5624 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677436 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677440 5624 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677443 5624 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677450 5624 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677454 5624 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677458 5624 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677461 5624 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677463 5624 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677467 5624 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677471 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677474 5624 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677477 5624 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677481 5624 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677485 5624 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677488 5624 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677491 5624 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677494 5624 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677500 5624 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:45.693510 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677503 5624 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677506 5624 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677511 5624 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677514 5624 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677517 5624 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677520 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677523 5624 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677526 5624 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677529 5624 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677535 5624 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677541 5624 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677545 5624 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677548 5624 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677551 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677554 5624 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677556 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677560 5624 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677564 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677568 5624 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:45.694503 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677573 5624 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677576 5624 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677580 5624 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677584 5624 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677589 5624 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677593 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677597 5624 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677601 5624 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677604 5624 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677607 5624 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677611 5624 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677615 5624 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677617 5624 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677620 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677623 5624 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677626 5624 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677632 5624 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677635 5624 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677638 5624 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677643 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:45.695293 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677647 5624 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677650 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677654 5624 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677657 5624 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677660 5624 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:01:45.677663 5624 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.677782 5624 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.679460 5624 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.679596 5624 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.679894 5624 server.go:1014] "Starting client certificate rotation" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.680096 5624 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.680195 5624 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:55:33 +0000 UTC" deadline="2028-04-17 17:01:20.571488262 +0000 UTC" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.680245 5624 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12838h59m34.891245567s" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.680682 5624 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.682735 5624 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.685513 5624 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:01:45.696741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.689011 5624 log.go:25] "Validated CRI v1 image API" Oct 30 18:01:45.697444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.690630 5624 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:01:45.697444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.695484 5624 fs.go:135] Filesystem UUIDs: map[16923d44-8629-470f-839d-6b4f569293ae:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f066eb9b-c585-46d3-ae10-ce61e8c725d9:/dev/nvme0n1p3] Oct 30 18:01:45.697598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.695551 5624 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/013f758ad4f0ef581b673c828693c4f15f85ae610e932510dcfc200c774ef0f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/013f758ad4f0ef581b673c828693c4f15f85ae610e932510dcfc200c774ef0f0/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/065d6a58c6b07132554588819641138002ac6008325a7c236aeebc0d6802e3f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/065d6a58c6b07132554588819641138002ac6008325a7c236aeebc0d6802e3f8/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bd29c553ab4bdfb87cf14071818c01fe1b7ce0c729a3e3fc690309609850121/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bd29c553ab4bdfb87cf14071818c01fe1b7ce0c729a3e3fc690309609850121/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f88e2dc7b92290621e92caf53d9bad2eef53ef513ba6b898462bc3a83fbbbc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f88e2dc7b92290621e92caf53d9bad2eef53ef513ba6b898462bc3a83fbbbc6/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51c430ef6903b5afe43d542f5ea3b5cf6376b8aea0ed810bdf98e6f784c019df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51c430ef6903b5afe43d542f5ea3b5cf6376b8aea0ed810bdf98e6f784c019df/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51c55f753b6c9d1b18977833eff0aad36398a4957953a5f84968a3e64eccef18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51c55f753b6c9d1b18977833eff0aad36398a4957953a5f84968a3e64eccef18/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/543d5806760cc4d4ff67ee214ae680634507a34462f7569d91186dab578caa5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/543d5806760cc4d4ff67ee214ae680634507a34462f7569d91186dab578caa5d/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76cd0ddb63e24298358d552efef0439c43d2191ce5d7da808491f211704edfcd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76cd0ddb63e24298358d552efef0439c43d2191ce5d7da808491f211704edfcd/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81e8fd7cdf67d35f08627b2453021726776de731591739cf1002aa10902d0037/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81e8fd7cdf67d35f08627b2453021726776de731591739cf1002aa10902d0037/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9075e2b792fc56fcf231b5d273c03e06d6c861f5aeee36057217024b25909971/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9075e2b792fc56fcf231b5d273c03e06d6c861f5aeee36057217024b25909971/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9359de080b2ab89cf29d6cec50873807f321c40c3a94475298aa812dd4120f93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9359de080b2ab89cf29d6cec50873807f321c40c3a94475298aa812dd4120f93/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96906cdfd91d34ffc8ba2c2349383c5d85b2bc9d27cef530921e2c112027f025/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96906cdfd91d34ffc8ba2c2349383c5d85b2bc9d27cef530921e2c112027f025/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0fa182c61696ac9605b93fd674d0ad614bb1310372893f63ce1f4bb4b020271/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0fa182c61696ac9605b93fd674d0ad614bb1310372893f63ce1f4bb4b020271/userdata/shm major:0 minor:417 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad1c70e922ad2d188d1e2a2b376e1dd72574b2fa11f4dcc332c49a174339416f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad1c70e922ad2d188d1e2a2b376e1dd72574b2fa11f4dcc332c49a174339416f/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2e2891508831f996f72384f06140e6016f74c014294ca16fd8d21e9eea7a8bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2e2891508831f996f72384f06140e6016f74c014294ca16fd8d21e9eea7a8bb/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b69283a81066adf3eba34d6a9de0e1812fc164e2ed5c4905fef813c09c763af2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b69283a81066adf3eba34d6a9de0e1812fc164e2ed5c4905fef813c09c763af2/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c16fe87eb3e0bcbd122feeb1530c976da6144525f3645851b01a6029bafcc5a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c16fe87eb3e0bcbd122feeb1530c976da6144525f3645851b01a6029bafcc5a1/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c627932e68de4b280509b3b4a67ba6ec8dba4dc843e0bfb4d1421cc4b8c8124a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c627932e68de4b280509b3b4a67ba6ec8dba4dc843e0bfb4d1421cc4b8c8124a/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cab59a04bf4c6c1297a19aecc6b8646be0695e1afe452bea91001a6c0de18f5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cab59a04bf4c6c1297a19aecc6b8646be0695e1afe452bea91001a6c0de18f5e/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d69e4d316b1946a705f8a7990d30cd7e8a19dcd97b6a095003e7217f9aa96b05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d69e4d316b1946a705f8a7990d30cd7e8a19dcd97b6a095003e7217f9aa96b05/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc0d8ffb48b2edb299cb70782535f4139f77ed4fef2addac6865033eb11f0f3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc0d8ffb48b2edb299cb70782535f4139f77ed4fef2addac6865033eb11f0f3d/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff6b759ead661c112c41275ddf4276b3b0cf22b4715d6e79fb0799755a93f4ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff6b759ead661c112c41275ddf4276b3b0cf22b4715d6e79fb0799755a93f4ca/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bbee04d-3f06-4ce5-8020-4d52b9715887/volumes/kubernetes.io~projected/kube-api-access-vrg4j:{mountpoint:/var/lib/kubelet/pods/0bbee04d-3f06-4ce5-8020-4d52b9715887/volumes/kubernetes.io~projected/kube-api-access-vrg4j major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bbee04d-3f06-4ce5-8020-4d52b9715887/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0bbee04d-3f06-4ce5-8020-4d52b9715887/volumes/kubernetes.io~secret/metrics-certs major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/216e2c3d-4562-43b7-a06b-30743fa0cf0c/volumes/kubernetes.io~projected/kube-api-access-dnt9h:{mountpoint:/var/lib/kubelet/pods/216e2c3d-4562-43b7-a06b-30743fa0cf0c/volumes/kubernetes.io~projected/kube-api-access-dnt9h major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/216e2c3d-4562-43b7-a06b-30743fa0cf0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/216e2c3d-4562-43b7-a06b-30743fa0cf0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~projected/kube-api-access-gxttb:{mountpoint:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~projected/kube-api-access-gxttb major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~projected/kube-api-access-vntf6:{mountpoint:/var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~projected/kube-api-access-vntf6 major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b/volumes/kubernetes.io~projected/kube-api-access-chz5j:{mountpoint:/var/lib/kubelet/pods/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b/volumes/kubernetes.io~projected/kube-api-access-chz5j major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b8bc620-2f0c-4060-b76e-2b2365e3621f/volumes/kubernetes.io~projected/kube-api-access-6krrq:{mountpoint:/var/lib/kubelet/pods/3b8bc620-2f0c-4060-b76e-2b2365e3621f/volumes/kubernetes.io~projected/kube-api-access-6krrq major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d05019b-efc6-444a-986d-6f63420ba3ff/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5d05019b-efc6-444a-986d-6f63420ba3ff/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~projected/kube-api-access-qw7kw:{mountpoint:/var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~projected/kube-api-access-qw7kw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/630626e4-d3eb-4e14-8c9d-9363718e5049/volumes/kubernetes.io~projected/kube-api-access-4vz24:{mountpoint:/var/lib/kubelet/pods/630626e4-d3eb-4e14-8c9d-9363718e5049/volumes/kubernetes.io~projected/kube-api-access-4vz24 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/630626e4-d3eb-4e14-8c9d-9363718e5049/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/630626e4-d3eb-4e14-8c9d-9363718e5049/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e4312f3-14a2-49b4-a6f1-4484d94a2217/volumes/kubernetes.io~projected/kube-api-access-8pmv5:{mountpoint:/var/lib/kubelet/pods/7e4312f3-14a2-49b4-a6f1-4484d94a2217/volumes/kubernetes.io~projected/kube-api-access-8pmv5 major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volumes/kubernetes.io~projected/kube-api-access-54gd4:{mountpoint:/var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volumes/kubernetes.io~projected/kube-api-access-54gd4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ca1d631-c3c1-4e4f-910c-76a1310a327c/volumes/kubernetes.io~projected/kube-api-access-vph5m:{mountpoint:/var/lib/kubelet/pods/8ca1d631-c3c1-4e4f-910c-76a1310a327c/volumes/kubernetes.io~projected/kube-api-access-vph5m major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e9cf883-7d59-46f0-b457-f84c7c678251/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8e9cf883-7d59-46f0-b457-f84c7c678251/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62ca62-c183-4859-8d52-a52aa8182c73/volumes/kubernetes.io~projected/kube-api-access-ch8nm:{mountpoint:/var/lib/kubelet/pods/9e62ca62-c183-4859-8d52-a52aa8182c73/volumes/kubernetes.io~projected/kube-api-access-ch8nm major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e62ca62-c183-4859-8d52-a52aa8182c73/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/9e62ca62-c183-4859-8d52-a52aa8182c73/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1cd5923-8633-4b24-8310-e06ac0a2de68/volumes/kubernetes.io~projected/kube-api-access-nlhg7:{mountpoint:/var/lib/kubelet/pods/b1cd5923-8633-4b24-8310-e06ac0a2de68/volumes/kubernetes.io~projected/kube-api-access-nlhg7 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1cd5923-8633-4b24-8310-e06ac0a2de68/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b1cd5923-8633-4b24-8310-e06ac0a2de68/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~projected/kube-api-access-z2h77:{mountpoint:/var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~projected/kube-api-access-z2h77 major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2/volumes/kubernetes.io~projected/kube-api-access-dx7z4:{mountpoint:/var/lib/kubelet/pods/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2/volumes/kubernetes.io~projected/kube-api-access-dx7z4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/config-reloader/2 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/prometheus/15 major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~empty-dir/config-out major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~projected/kube-api-access-vbjpt:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~projected/kube-api-access-vbjpt major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~projected/tls-assets major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/config major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/web-config major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4acf9b8-e56f-4958-9742-7303e972ae20/volumes/kubernetes.io~projected/kube-api-access-qqqpm:{mountpoint:/var/lib/kubelet/pods/d4acf9b8-e56f-4958-9742-7303e972ae20/volumes/kubernetes.io~projected/kube-api-access-qqqpm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7906b48-2326-4074-a2a4-a62654c2b785/volumes/kubernetes.io~projected/kube-api-access-gn4qt:{mountpoint:/var/lib/kubelet/pods/d7906b48-2326-4074-a2a4-a62654c2b785/volumes/kubernetes.io~projected/kube-api-access-gn4qt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~projected/kube-api-access-5bh65:{mountpoint:/var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~projected/kube-api-access-5bh65 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:250 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7bff4f1108f9a9656d05ae2480a60832529b88f1cf3afcf26298c44fcb3f27fa/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d4b783a1b8e3f1b46d8a8431585631bcdedf8d04f97eb990109884d6580ee9a1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8d8dd4dfd6e6f05ac9a4438694818374597181a825115784a07589e6c0e95eed/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8969865656f3894599ea399c33d612fdbde47c7d8236251e1a8d1347c7fe4d79/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d2ada5068a38c5b5cacf8c651cc51d8124c72f438216acd9cac8348cb55c3f82/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/61c4e733613f4858dd3f74f2540d02a13c5278cdd6e5d5887e7608d77c3689ae/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/08693b9bb7161ed7669532c2fa14bb7da20a945e6bfa77b4e47cc6033d498785/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2a7c4743d082a0acf514b182c3ca111c790d58d360ed6280a80e1b0546440a93/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/17d69a7ff648251bd4f92aeddace28b2956ff9f951413e6d23a9b2bc7adcb758/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/559c5f853094c602f3c4ec3a62a375311ac70c4e10ea8801d53491e01615f5a9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/41b9f684346cc0b5d9aa0daf9968d36ef2b97c0143e9fabee9648a7447a3af0b/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/9290d17d80d3dc7a7028170516b0086d4eaf7d586d68eaa1453df7f0b8f11f11/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/0a951e90a8bd2bf9c126e52a63493f22ed1db4d871aa05916e197a7ed2a23140/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2a2705890491f09dd5fc2c7ac7bd5be118138b8667a77a705e395706a641f882/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/dae8543d84ef1e778cd8839483750460ef5f07939bc64c21aebadb029568d048/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/3e9872a53c61296ace31fec90fe5f3bf967f729bfd4c8470736e623331d49f2b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/fe9907b86bec1b07f87b1fbe70644f884d8d760a78a1910776f92eede7965b68/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/cce3fecf6b2e83604a8dfb551fc71e3a777a73065258bd85cea2b73f8bcf6c3e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1ce4bc5a2a021f34a66a19ad75d69a3a31562fbcb37cb372f88dc8030ee41f65/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/671641c31469b3b72a6fae2b2ee783d281ce8af49ab27cd0170a1f974e2ae2a3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/141994977e73dfa3cb8e7461ac8340e93a36b211c7c059c8b50bab1391a8568a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/db483121f5401c9b2e284ca1dbff408aa0c90b2e2df1fbcfd2ee61cf3798d2f1/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/47e15e698acd636cd1c396e3703cfe544f6c9befde22af6f32342c8afdc902ce/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/5c7d5fc91acd39ee8666c9634e98db239a723080235cc5f9950828ddb2137003/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/5a10c1ad72c6b521ad8786f6221d48d7531d6772e724ece430d97103803473b3/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/bcdda5050090b4ba01b2114b29d794e60f93986ff81ff23b787d010fb7a5deb6/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/b71cf41f13e2d19e5ce54a0e23a4c1a0e9dfd3e9d6420b8e588b1c365e436817/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/f4c3adad6d3cf1e65bfd392d4280b1ed132b34bfb73c9d7ffffeb1938e198969/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/bb884c8108432ccb0107aa6ce693a44e003156d30e9b09b61126c2d8fbad5fe3/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/eeaacc01962eb3307136905d23b55200f43fc55ae087fecfec98f2a1a5822569/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/ea1df2c16c1f59772bea86b36a0efa2591f6fa18f337db2cd37b2382f7c74f27/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/52bff574a802feeba8f482704af25e006e67483ab0dc28f1584b163500d7260a/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/b1b9c1884eb73efff58ab50a1e3d9a6588147e0ade902919ef9794d5ae631b82/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/1da30ce3ca295ef90baebfaca35e668f9e212adf0af0487809fa4901baf013b1/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/9f75164055c42cfd72b13391003d82a2cf1bab1d98be2643a5ba38b2b2e6021f/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/d6fbf8e18504f89bfb2be384751f6a8454d4319209800b3d7ad7b4023228ee60/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/f97ff2e1f692e2234d65700480a82839462a25f286bf75a64281406d295c443f/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/d7f72d0740842bb372ac14ca817fa4850b2f2faf82608e2ef0b90b9892bb779d/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/74d65b5a1a34d0416c805375d95e483f7658968e4cf1c1876f1ee39f4a159b52/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/4adc3357655d4f5b821a7376d00136327aeb27b93dccc75c0049957a5eaa77d0/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/67eb067ac534d92b968e5556159e9b637ea6ea6cd07c68ae60f09111feb11c60/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/94feb293f64caec82d1e649abdae78afd04c62e76e1384575b7f59399cdfe0e9/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/297d23a946b3372caab2eeb623b6d840201e606bed11c2289b5cec6c1f684c31/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/e5a8ee662691e0aab1ed0508976e286594b6478232949048561d5bbb27ea12fa/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/134f2bd157401b25adbc7e733bbaf39965063d9ebe09d941e270f1cc855bc298/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/b402f4057d8b6277d783da2f834093ce011c067fe2b38dca32991eb13c9b7ba9/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/ef910d0556383b40930ff25df7fcf373398dbd16a01682dc0f0a4db5ed2d0c87/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/214b0b0221b0b059fb80d0535233ea2e5941ea85f228cab6eeec53a53fd85a4a/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/15f6dd5875d1bc8d673d58787ea8ddba4fbcc3dd3053a52e243c507da822927b/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/c5fdf054c38ff2d9a2ec25d0d527865a75168dcec3377729d32305eb91186e78/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/7ccb9de08ce66f6c79cba13c6d44590b4fd4b9f1fde2da6294b8e333efe611e9/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/f19a575f482af0221ac91cc1d64acaed089361ac7d3c32203bf1dc75901afe53/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/4e38a2c12afe494fbeae832c768215c615f77b85d29eec3f5e12c37955bf9da8/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/aadf0ead029d49cdec8aec2cdae2d5b8f924749c427f0abe35fb59dae0763c2c/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/58d68334c1adc827443f23126988ff90c7b2253f13b428930f96e43b4bc42066/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/04b8a1a025f3335492cbeaf0560aafbf52a1ac0d5f809d30f745473e61acf522/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/4f5139cf3b77e595b77d7e07dd69372f34135e876ac876eec1e12a6c6a217280/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/5748f950d46d2315d662fec37c23d19a6ccb2cd98084c1f0e6d6754b8786ce60/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/dfd236999b5e41f382e314230d4abe46e0d07c7102a2fe11218267e84fbe6c22/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/8cce1f8761993f71a68543f02b7bf132d0e8f5372acdceb9a279c91b95f662c8/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/90c4e1aaf7bc7870e206afd56d8c262c3f222198cdbc58b4b57f79fb6ffa9e03/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/cbf3878438e295b5dfe37adbcb284c80f997bf64d0d0244e946e283f559634aa/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/5da336dcb87bc47551d10e8bd29c2ae0cd01197fc1319abbac409a1d87638e30/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/42d8a5c4fcaaf9dec7cf47bcedf411a5a601475b3ffbd7b4fc6a95c78f681f46/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/97767e277bab7edd5b370507a603b880ce287ab8ec65d765f6da89859006ee00/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/29887f62692542a690ae41dae8b23c41cb6b58f4650d2b057a343f4c6a6627e1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/587c9cbf54540fbd170bc20899c3bde4ea5eca44e828e5fabdaae3790bed576f/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/380c1f45cd3eefa2d60b3666bd90e11e4f044fbcb010374c8290ec5fa88d8160/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/e3f6ab8681c737813ae64e9f2fd017969814388c13993a7a4bc584304f49226b/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/736b6c81467718f644fe1ec24a0e392a86e05ac10d5b14095c4cb14e81f1d6ee/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/15c7f82895cd3178e9ab6783bd591807cbde465220b61620d11d30611ee6ad78/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/c66864fb92d3df736a1fdacd5cc5377f5989e873094e8072046ee39a2d94d834/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/1779a85688cfa20b308a5084e612e487262db4582fba807018e3595068c2efa0/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e36344b52129cb4cd77aa9721fc67e2ff4433c0ed105a746a2289821dea9a778/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 18:01:45.709704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.709115 5624 manager.go:217] Machine: {Timestamp:2026-10-30 18:01:45.707749178 +0000 UTC m=+0.096277333 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100973 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9ef5af269a4be26c0f656ad3a0c6 SystemUUID:ec2a9ef5-af26-9a4b-e26c-0f656ad3a0c6 BootID:a8c99ca4-873b-4c23-9c9f-82e1f1d2eed2 Filesystems:[{Device:/var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62ca62-c183-4859-8d52-a52aa8182c73/volumes/kubernetes.io~projected/kube-api-access-ch8nm DeviceMajor:0 DeviceMinor:325 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c627932e68de4b280509b3b4a67ba6ec8dba4dc843e0bfb4d1421cc4b8c8124a/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c16fe87eb3e0bcbd122feeb1530c976da6144525f3645851b01a6029bafcc5a1/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e9cf883-7d59-46f0-b457-f84c7c678251/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:411 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~projected/kube-api-access-gxttb DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bd29c553ab4bdfb87cf14071818c01fe1b7ce0c729a3e3fc690309609850121/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~projected/kube-api-access-vbjpt DeviceMajor:0 DeviceMinor:357 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/013f758ad4f0ef581b673c828693c4f15f85ae610e932510dcfc200c774ef0f0/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d69e4d316b1946a705f8a7990d30cd7e8a19dcd97b6a095003e7217f9aa96b05/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~projected/kube-api-access-5bh65 DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1cd5923-8633-4b24-8310-e06ac0a2de68/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/630626e4-d3eb-4e14-8c9d-9363718e5049/volumes/kubernetes.io~projected/kube-api-access-4vz24 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9075e2b792fc56fcf231b5d273c03e06d6c861f5aeee36057217024b25909971/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2e2891508831f996f72384f06140e6016f74c014294ca16fd8d21e9eea7a8bb/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0fa182c61696ac9605b93fd674d0ad614bb1310372893f63ce1f4bb4b020271/userdata/shm DeviceMajor:0 DeviceMinor:417 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2/volumes/kubernetes.io~projected/kube-api-access-dx7z4 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9e62ca62-c183-4859-8d52-a52aa8182c73/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:321 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:353 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cab59a04bf4c6c1297a19aecc6b8646be0695e1afe452bea91001a6c0de18f5e/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~projected/kube-api-access-vntf6 DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96906cdfd91d34ffc8ba2c2349383c5d85b2bc9d27cef530921e2c112027f025/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51c430ef6903b5afe43d542f5ea3b5cf6376b8aea0ed810bdf98e6f784c019df/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bbee04d-3f06-4ce5-8020-4d52b9715887/volumes/kubernetes.io~projected/kube-api-access-vrg4j DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b1cd5923-8633-4b24-8310-e06ac0a2de68/volumes/kubernetes.io~projected/kube-api-access-nlhg7 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/216e2c3d-4562-43b7-a06b-30743fa0cf0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:347 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:352 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81e8fd7cdf67d35f08627b2453021726776de731591739cf1002aa10902d0037/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volumes/kubernetes.io~projected/kube-api-access-54gd4 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f88e2dc7b92290621e92caf53d9bad2eef53ef513ba6b898462bc3a83fbbbc6/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc0d8ffb48b2edb299cb70782535f4139f77ed4fef2addac6865033eb11f0f3d/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51c55f753b6c9d1b18977833eff0aad36398a4957953a5f84968a3e64eccef18/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3275088d-42df-41bd-8b42-b739a353962d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:350 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/216e2c3d-4562-43b7-a06b-30743fa0cf0c/volumes/kubernetes.io~projected/kube-api-access-dnt9h DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b/volumes/kubernetes.io~projected/kube-api-access-chz5j DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/630626e4-d3eb-4e14-8c9d-9363718e5049/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/065d6a58c6b07132554588819641138002ac6008325a7c236aeebc0d6802e3f8/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b69283a81066adf3eba34d6a9de0e1812fc164e2ed5c4905fef813c09c763af2/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9359de080b2ab89cf29d6cec50873807f321c40c3a94475298aa812dd4120f93/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5d05019b-efc6-444a-986d-6f63420ba3ff/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d4acf9b8-e56f-4958-9742-7303e972ae20/volumes/kubernetes.io~projected/kube-api-access-qqqpm DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d7906b48-2326-4074-a2a4-a62654c2b785/volumes/kubernetes.io~projected/kube-api-access-gn4qt DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b8bc620-2f0c-4060-b76e-2b2365e3621f/volumes/kubernetes.io~projected/kube-api-access-6krrq DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad1c70e922ad2d188d1e2a2b376e1dd72574b2fa11f4dcc332c49a174339416f/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76cd0ddb63e24298358d552efef0439c43d2191ce5d7da808491f211704edfcd/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:349 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff6b759ead661c112c41275ddf4276b3b0cf22b4715d6e79fb0799755a93f4ca/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e4312f3-14a2-49b4-a6f1-4484d94a2217/volumes/kubernetes.io~projected/kube-api-access-8pmv5 DeviceMajor:0 DeviceMinor:416 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/8ca1d631-c3c1-4e4f-910c-76a1310a327c/volumes/kubernetes.io~projected/kube-api-access-vph5m DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/269d1132-4f1d-4b95-b545-2b2659d245a7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a6cc3b-391d-4626-908a-5466994e21d5/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:250 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf49c75c-fb11-4c2d-ac07-b41d85f8f106/volumes/kubernetes.io~projected/kube-api-access-z2h77 DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0bbee04d-3f06-4ce5-8020-4d52b9715887/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:415 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/82127ba1-4bef-43cd-8550-719a4402fcab/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5f811312-1c2a-4099-9cbe-efe1c5063b66/volumes/kubernetes.io~projected/kube-api-access-qw7kw DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/543d5806760cc4d4ff67ee214ae680634507a34462f7569d91186dab578caa5d/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:065d6a58c6b0713 MacAddress:5e:cb:e7:14:a2:ee Speed:10000 Mtu:8901} {Name:4bd29c553ab4bdf MacAddress:9a:0f:33:a4:99:af Speed:10000 Mtu:8901} {Name:4f88e2dc7b92290 MacAddress:a6:66:d7:cc:f9:16 Speed:10000 Mtu:8901} {Name:51c55f753b6c9d1 MacAddress:1e:f1:9f:89:74:3f Speed:10000 Mtu:8901} {Name:9075e2b792fc56f MacAddress:a6:c6:b3:a2:9c:ea Speed:10000 Mtu:8901} {Name:a0fa182c61696ac MacAddress:1a:30:c7:fa:b8:e6 Speed:10000 Mtu:8901} {Name:b2e2891508831f9 MacAddress:5a:f8:58:50:02:5e Speed:10000 Mtu:8901} {Name:b69283a81066adf MacAddress:f6:3a:6a:66:fd:01 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e8:25:51:69 Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:18:d3:05:6d:92 Speed:0 Mtu:8901} {Name:c627932e68de4b2 MacAddress:de:ea:71:c3:cf:b0 Speed:10000 Mtu:8901} {Name:cab59a04bf4c6c1 MacAddress:d6:ee:7c:a2:1a:53 Speed:10000 Mtu:8901} {Name:d69e4d316b1946a MacAddress:b2:16:24:a1:a7:a9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e8:25:51:69 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0e:45:52:28:ba:c0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:91:fa:f4:7a:73 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:01:45.709704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.709699 5624 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:01:45.710402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.709815 5624 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:01:45.710612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.710531 5624 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:01:45.710879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.710614 5624 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-47.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:01:45.711012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.710890 5624 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:01:45.711012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.710907 5624 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:01:45.711012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.710962 5624 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:01:45.711012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.711002 5624 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:01:45.711325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.711301 5624 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:01:45.711756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.711729 5624 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:01:45.712068 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.712045 5624 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:01:45.712171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.712133 5624 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:01:45.712171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.712158 5624 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:01:45.712262 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.712182 5624 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:01:45.712262 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.712203 5624 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:01:45.712262 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.712250 5624 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:01:45.713472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.713428 5624 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:01:45.713472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.713461 5624 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:01:45.716420 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.716372 5624 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:01:45.720605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.720544 5624 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:01:45.720605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.720589 5624 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:01:45.720793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.720781 5624 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 18:01:45.721155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721134 5624 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:01:45.721203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721175 5624 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:01:45.721414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721401 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:01:45.721475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721425 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:01:45.721475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721435 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:01:45.721475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721441 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:01:45.721475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721449 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:01:45.721475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721457 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:01:45.721475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721465 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:01:45.721475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721472 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:01:45.721685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721481 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:01:45.721685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721489 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:01:45.721685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721511 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:01:45.721685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721523 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:01:45.721685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721565 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:01:45.721685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.721572 5624 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:01:45.727951 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.727893 5624 server.go:1263] "Started kubelet" Oct 30 18:01:45.728104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.728025 5624 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:01:45.728284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.728151 5624 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:01:45.728353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.728339 5624 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:01:45.729090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.728962 5624 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:01:45.729274 ip-10-0-133-47 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:01:45.730834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.730717 5624 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:01:45.739158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.739106 5624 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.739370 5624 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.739448 5624 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.739465 5624 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.739464 5624 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:35 +0000 UTC" deadline="2028-07-04 13:29:13.000815188 +0000 UTC" Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.739497 5624 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14707h27m27.261321752s" Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:45.739584 5624 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-47.ec2.internal\" not found" Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.739664 5624 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.741983 5624 factory.go:55] Registering systemd factory Oct 30 18:01:45.742266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.742037 5624 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:01:45.743413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743362 5624 factory.go:153] Registering CRI-O factory Oct 30 18:01:45.743413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743416 5624 factory.go:223] Registration of the crio container factory successfully Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743406 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-grpc-tls" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743501 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b8bc620-2f0c-4060-b76e-2b2365e3621f" volumeName="kubernetes.io/empty-dir/3b8bc620-2f0c-4060-b76e-2b2365e3621f-tmp-dir" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743521 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d05019b-efc6-444a-986d-6f63420ba3ff" volumeName="kubernetes.io/secret/5d05019b-efc6-444a-986d-6f63420ba3ff-agent-certs" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743539 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f811312-1c2a-4099-9cbe-efe1c5063b66" volumeName="kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-tmp" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743555 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82127ba1-4bef-43cd-8550-719a4402fcab" volumeName="kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-script-lib" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743568 5624 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743574 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cd5923-8633-4b24-8310-e06ac0a2de68" volumeName="kubernetes.io/secret/b1cd5923-8633-4b24-8310-e06ac0a2de68-metrics-tls" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743593 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2" volumeName="kubernetes.io/configmap/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-serviceca" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743608 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743633 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743653 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ca1d631-c3c1-4e4f-910c-76a1310a327c" volumeName="kubernetes.io/projected/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kube-api-access-vph5m" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743663 5624 factory.go:103] Registering Raw factory Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743668 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62ca62-c183-4859-8d52-a52aa8182c73" volumeName="kubernetes.io/projected/9e62ca62-c183-4859-8d52-a52aa8182c73-kube-api-access-ch8nm" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743686 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743704 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743720 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743744 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4acf9b8-e56f-4958-9742-7303e972ae20" volumeName="kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-binary-copy" seLinuxMountContext="" Oct 30 18:01:45.743774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743769 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743790 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f811312-1c2a-4099-9cbe-efe1c5063b66" volumeName="kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-tuned" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743839 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39764b0f-c4d1-4758-b8c2-9fb4d001ed8b" volumeName="kubernetes.io/projected/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-kube-api-access-chz5j" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743862 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743897 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bbee04d-3f06-4ce5-8020-4d52b9715887" volumeName="kubernetes.io/projected/0bbee04d-3f06-4ce5-8020-4d52b9715887-kube-api-access-vrg4j" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743919 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3275088d-42df-41bd-8b42-b739a353962d" volumeName="kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743960 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3275088d-42df-41bd-8b42-b739a353962d" volumeName="kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-tls" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743988 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39764b0f-c4d1-4758-b8c2-9fb4d001ed8b" volumeName="kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cni-binary-copy" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744007 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82127ba1-4bef-43cd-8550-719a4402fcab" volumeName="kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-config" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744025 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf49c75c-fb11-4c2d-ac07-b41d85f8f106" volumeName="kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744040 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a6cc3b-391d-4626-908a-5466994e21d5" volumeName="kubernetes.io/empty-dir/e2a6cc3b-391d-4626-908a-5466994e21d5-volume-directive-shadow" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744061 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/projected/269d1132-4f1d-4b95-b545-2b2659d245a7-kube-api-access-gxttb" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744079 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3275088d-42df-41bd-8b42-b739a353962d" volumeName="kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744102 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82127ba1-4bef-43cd-8550-719a4402fcab" volumeName="kubernetes.io/projected/82127ba1-4bef-43cd-8550-719a4402fcab-kube-api-access-54gd4" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744119 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf49c75c-fb11-4c2d-ac07-b41d85f8f106" volumeName="kubernetes.io/projected/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-kube-api-access-z2h77" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744133 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744159 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a6cc3b-391d-4626-908a-5466994e21d5" volumeName="kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744176 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744197 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744212 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744228 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d05019b-efc6-444a-986d-6f63420ba3ff" volumeName="kubernetes.io/configmap/5d05019b-efc6-444a-986d-6f63420ba3ff-konnectivity-ca" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744243 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="630626e4-d3eb-4e14-8c9d-9363718e5049" volumeName="kubernetes.io/secret/630626e4-d3eb-4e14-8c9d-9363718e5049-cert" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744263 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216e2c3d-4562-43b7-a06b-30743fa0cf0c" volumeName="kubernetes.io/projected/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-api-access-dnt9h" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744282 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3275088d-42df-41bd-8b42-b739a353962d" volumeName="kubernetes.io/empty-dir/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-textfile" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744297 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e4312f3-14a2-49b4-a6f1-4484d94a2217" volumeName="kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744312 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82127ba1-4bef-43cd-8550-719a4402fcab" volumeName="kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-env-overrides" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744329 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4acf9b8-e56f-4958-9742-7303e972ae20" volumeName="kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744343 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7906b48-2326-4074-a2a4-a62654c2b785" volumeName="kubernetes.io/projected/d7906b48-2326-4074-a2a4-a62654c2b785-kube-api-access-gn4qt" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744358 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a6cc3b-391d-4626-908a-5466994e21d5" volumeName="kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-metrics-client-ca" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.743690 5624 manager.go:1199] Started watching for new ooms in manager Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744374 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a6cc3b-391d-4626-908a-5466994e21d5" volumeName="kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744433 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216e2c3d-4562-43b7-a06b-30743fa0cf0c" volumeName="kubernetes.io/configmap/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744447 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744485 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3275088d-42df-41bd-8b42-b739a353962d" volumeName="kubernetes.io/projected/3275088d-42df-41bd-8b42-b739a353962d-kube-api-access-vntf6" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744501 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cd5923-8633-4b24-8310-e06ac0a2de68" volumeName="kubernetes.io/configmap/b1cd5923-8633-4b24-8310-e06ac0a2de68-config-volume" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744576 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744603 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a6cc3b-391d-4626-908a-5466994e21d5" volumeName="kubernetes.io/projected/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-api-access-5bh65" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744627 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216e2c3d-4562-43b7-a06b-30743fa0cf0c" volumeName="kubernetes.io/secret/216e2c3d-4562-43b7-a06b-30743fa0cf0c-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744653 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/empty-dir/269d1132-4f1d-4b95-b545-2b2659d245a7-tmp" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744678 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744697 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/configmap/269d1132-4f1d-4b95-b545-2b2659d245a7-metrics-client-ca" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744720 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3275088d-42df-41bd-8b42-b739a353962d" volumeName="kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-metrics-client-ca" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744776 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744798 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4acf9b8-e56f-4958-9742-7303e972ae20" volumeName="kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 18:01:45.744766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744816 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744837 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b8bc620-2f0c-4060-b76e-2b2365e3621f" volumeName="kubernetes.io/projected/3b8bc620-2f0c-4060-b76e-2b2365e3621f-kube-api-access-6krrq" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744855 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f811312-1c2a-4099-9cbe-efe1c5063b66" volumeName="kubernetes.io/projected/5f811312-1c2a-4099-9cbe-efe1c5063b66-kube-api-access-qw7kw" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744875 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cd5923-8633-4b24-8310-e06ac0a2de68" volumeName="kubernetes.io/empty-dir/b1cd5923-8633-4b24-8310-e06ac0a2de68-tmp-dir" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744910 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cd5923-8633-4b24-8310-e06ac0a2de68" volumeName="kubernetes.io/projected/b1cd5923-8633-4b24-8310-e06ac0a2de68-kube-api-access-nlhg7" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744952 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744972 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.744996 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745016 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216e2c3d-4562-43b7-a06b-30743fa0cf0c" volumeName="kubernetes.io/empty-dir/216e2c3d-4562-43b7-a06b-30743fa0cf0c-data-volume" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745034 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2" volumeName="kubernetes.io/projected/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-kube-api-access-dx7z4" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745056 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745075 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4acf9b8-e56f-4958-9742-7303e972ae20" volumeName="kubernetes.io/projected/d4acf9b8-e56f-4958-9742-7303e972ae20-kube-api-access-qqqpm" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745084 5624 manager.go:319] Starting recovery of all containers Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745094 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7906b48-2326-4074-a2a4-a62654c2b785" volumeName="kubernetes.io/configmap/d7906b48-2326-4074-a2a4-a62654c2b785-iptables-alerter-script" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745114 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a6cc3b-391d-4626-908a-5466994e21d5" volumeName="kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745129 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bbee04d-3f06-4ce5-8020-4d52b9715887" volumeName="kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745148 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39764b0f-c4d1-4758-b8c2-9fb4d001ed8b" volumeName="kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-daemon-config" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745169 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82127ba1-4bef-43cd-8550-719a4402fcab" volumeName="kubernetes.io/secret/82127ba1-4bef-43cd-8550-719a4402fcab-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745188 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e9cf883-7d59-46f0-b457-f84c7c678251" volumeName="kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745206 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745227 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745249 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745270 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269d1132-4f1d-4b95-b545-2b2659d245a7" volumeName="kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745288 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="630626e4-d3eb-4e14-8c9d-9363718e5049" volumeName="kubernetes.io/projected/630626e4-d3eb-4e14-8c9d-9363718e5049-kube-api-access-4vz24" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745309 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e62ca62-c183-4859-8d52-a52aa8182c73" volumeName="kubernetes.io/secret/9e62ca62-c183-4859-8d52-a52aa8182c73-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745328 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf49c75c-fb11-4c2d-ac07-b41d85f8f106" volumeName="kubernetes.io/configmap/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-metrics-client-ca" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745352 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf49c75c-fb11-4c2d-ac07-b41d85f8f106" volumeName="kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745371 5624 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc1afeda-d283-47f1-978a-43aeeae7d6cb" volumeName="kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745387 5624 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:01:45.747155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.745403 5624 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:01:45.774472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.774307 5624 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 18:01:45.774761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.774741 5624 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:01:45.776152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.776108 5624 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:01:45.779631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.779518 5624 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:01:45.782481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.782422 5624 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:01:45.782481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.782478 5624 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:01:45.782888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.782514 5624 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:01:45.782888 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:45.782584 5624 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 18:01:45.798404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.798350 5624 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:01:45.798816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.798506 5624 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:01:45.827551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.827477 5624 generic.go:356] "Generic (PLEG): container finished" podID="9bfefcd36786ca02a8752b4bece34acc" containerID="10db0fe1a1ec6c19b84ea7d63467105f64b720c15d8778bde9304338ea9fd028" exitCode=0 Oct 30 18:01:45.873423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.873355 5624 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="10e29837eeda3ba13ab7b3a11c102287a128a9e2b509629e973205e97d81d371" exitCode=0 Oct 30 18:01:45.873423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.873409 5624 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="ba97567dd1afd3c4d2a3b9881f911aa35f4c93cb0bb1957dbe709f0e74bd4c6b" exitCode=0 Oct 30 18:01:45.873423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.873418 5624 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="96d28d0887239ac5b3771203150b689afccdad44f9752eb21ec23f7f06768068" exitCode=0 Oct 30 18:01:45.873423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.873430 5624 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="f4d04c3962bbc760c948f7c24a7d21e5c7ae6fd6e84cc05e63afb858c016f074" exitCode=0 Oct 30 18:01:45.873423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.873437 5624 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="f0af9f0c40b697a4949d516b721801d2e5a3d9f1fb14a30de31d6e9fc6ea596a" exitCode=0 Oct 30 18:01:45.873423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.873444 5624 generic.go:356] "Generic (PLEG): container finished" podID="d4acf9b8-e56f-4958-9742-7303e972ae20" containerID="a99731a377b7328db31b8393afc3b127794ba1dda7d3ef4cb0edd6c7cf173dd1" exitCode=0 Oct 30 18:01:45.882724 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:45.882669 5624 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 18:01:45.886565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.886488 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerID="eeaa04d40d7f51db91196146c7868ea79da05945fb9a4f55784220ab9404ea8c" exitCode=0 Oct 30 18:01:45.895973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.895857 5624 generic.go:356] "Generic (PLEG): container finished" podID="3275088d-42df-41bd-8b42-b739a353962d" containerID="0c50409e0912bbf0ec61d39aebe58622a155b2a0638f0d006830ab1cf7f30917" exitCode=0 Oct 30 18:01:45.918038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.917947 5624 manager.go:324] Recovery completed Oct 30 18:01:45.956003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.955912 5624 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:01:45.956003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.955982 5624 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:01:45.956300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.956048 5624 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:01:45.956375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.956342 5624 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 18:01:45.956420 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.956379 5624 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 18:01:45.956452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.956423 5624 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 18:01:45.956452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.956435 5624 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 18:01:45.956452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.956448 5624 policy_none.go:50] "Start" Oct 30 18:01:45.956586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.956473 5624 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:01:45.956586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.956492 5624 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:01:45.957794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.957748 5624 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 18:01:45.957794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.957796 5624 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 18:01:45.957882 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.957807 5624 policy_none.go:44] "Start" Oct 30 18:01:45.965974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.965871 5624 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:01:45.977573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.977531 5624 manager.go:341] "Starting Device Plugin manager" Oct 30 18:01:45.978861 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:45.977605 5624 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:01:45.979020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.978876 5624 server.go:87] "Starting device plugin registration server" Oct 30 18:01:45.979487 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.979462 5624 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:01:45.979652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.979497 5624 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:01:45.979785 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.979753 5624 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:01:45.980051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.979988 5624 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:01:45.980213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.980076 5624 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:01:45.980213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.980086 5624 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:01:45.980351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.980147 5624 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:01:45.980042326Z","UUID":"be61f0db-087a-480b-b0fc-d19029654625","Handler":null,"Name":"","Endpoint":""} Oct 30 18:01:45.982503 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:45.982449 5624 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:01:45.982847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.982471 5624 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:01:45.982847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:45.982619 5624 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:01:46.080911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.080746 5624 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:01:46.083352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.083273 5624 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal"] Oct 30 18:01:46.084077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084038 5624 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:01:46.084284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084099 5624 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:01:46.084284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084114 5624 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:01:46.084284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084208 5624 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.084479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084428 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" event={"ID":"9bfefcd36786ca02a8752b4bece34acc","Type":"ContainerStarted","Data":"70a7fcc62bf6514a3536b602e13d70063402bb8d9dc22351c70a74e8d50d0e28"} Oct 30 18:01:46.084531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084498 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" event={"ID":"9bfefcd36786ca02a8752b4bece34acc","Type":"ContainerDied","Data":"10db0fe1a1ec6c19b84ea7d63467105f64b720c15d8778bde9304338ea9fd028"} Oct 30 18:01:46.084565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084528 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" event={"ID":"9bfefcd36786ca02a8752b4bece34acc","Type":"ContainerStarted","Data":"9359de080b2ab89cf29d6cec50873807f321c40c3a94475298aa812dd4120f93"} Oct 30 18:01:46.084565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084554 5624 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.084660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084636 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" event={"ID":"05806b328c870eba3d65f2b11ca3bf01","Type":"ContainerStarted","Data":"fbe6b579808550835f3e59759685199b7a201c7007d1cab5a714c4ed172dbc1f"} Oct 30 18:01:46.084705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.084674 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" event={"ID":"05806b328c870eba3d65f2b11ca3bf01","Type":"ContainerStarted","Data":"81e8fd7cdf67d35f08627b2453021726776de731591739cf1002aa10902d0037"} Oct 30 18:01:46.085054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.085033 5624 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.105657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.105593 5624 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:01:46.105848 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:46.105688 5624 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-47.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.105848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.105623 5624 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:01:46.105848 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:46.105792 5624 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.119600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.119541 5624 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.119919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.119707 5624 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.176528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.176409 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.176825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.176570 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.176825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.176657 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05806b328c870eba3d65f2b11ca3bf01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-47.ec2.internal\" (UID: \"05806b328c870eba3d65f2b11ca3bf01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.277434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.277345 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.277434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.277426 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.277434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.277451 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05806b328c870eba3d65f2b11ca3bf01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-47.ec2.internal\" (UID: \"05806b328c870eba3d65f2b11ca3bf01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.277855 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.277507 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05806b328c870eba3d65f2b11ca3bf01-config\") pod \"kube-apiserver-proxy-ip-10-0-133-47.ec2.internal\" (UID: \"05806b328c870eba3d65f2b11ca3bf01\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.277855 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.277513 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.277855 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.277516 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfefcd36786ca02a8752b4bece34acc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" (UID: \"9bfefcd36786ca02a8752b4bece34acc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.713189 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.713112 5624 apiserver.go:52] "Watching apiserver" Oct 30 18:01:46.763154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.763099 5624 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:01:46.764782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.764707 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9l55k","openshift-monitoring/thanos-querier-74f988c6b9-jw4rr","openshift-ovn-kubernetes/ovnkube-node-g74ls","kube-system/global-pull-secret-syncer-qg2lp","openshift-dns/node-resolver-n97j8","openshift-multus/multus-additional-cni-plugins-gdxmg","openshift-dns/dns-default-6mcgq","openshift-cluster-node-tuning-operator/tuned-k8mxn","openshift-monitoring/monitoring-plugin-77f4f76594-f87jc","openshift-image-registry/node-ca-l6vjs","openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29","openshift-multus/multus-sq5cb","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-hchmm","openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll","kube-system/konnectivity-agent-6w92c","openshift-insights/insights-runtime-extractor-x6tdb","kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal","openshift-network-diagnostics/network-check-target-xpbwl","openshift-monitoring/node-exporter-7bjq6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal","openshift-network-operator/iptables-alerter-fl2ts","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6"] Oct 30 18:01:46.780287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780224 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:01:46.780673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780332 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.780673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780384 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.780673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780470 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:01:46.780673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780492 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:01:46.780991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780743 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:01:46.780991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780806 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:01:46.780991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780844 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" Oct 30 18:01:46.780991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780846 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.780991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.780911 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:01:46.781275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.781213 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:01:46.781334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.781294 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:01:46.781465 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.781420 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" Oct 30 18:01:46.781465 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.781439 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" Oct 30 18:01:46.781727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.781529 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.781727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.781668 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.781825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.781737 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.781825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.781781 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:01:46.782131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.782108 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:01:46.782353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.782193 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" Oct 30 18:01:46.782353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.782253 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" Oct 30 18:01:46.789835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.789768 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:01:46.843187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.843107 5624 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:01:46.880327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880206 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:01:46.880327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880203 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:01:46.880327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880261 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:01:46.880327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880284 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.880327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880325 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-tuned\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880390 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880387 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-var-lib-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880489 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880501 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-node-log\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880569 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-tuned\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880665 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880699 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880711 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880709 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880843 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880871 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880892 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880960 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.880891 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-var-lib-kubelet\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881027 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-log-socket\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881047 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881064 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-k8s-cni-cncf-io\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881106 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881113 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881152 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881183 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881220 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-host\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881234 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881262 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-script-lib\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881294 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-binary-copy\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881328 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881386 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881506 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.881524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881533 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881640 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881756 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881830 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.881877 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-wtmp\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882000 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882161 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882176 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882254 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882285 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-conf\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882326 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-tmp\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882356 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-script-lib\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882366 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4qt\" (UniqueName: \"kubernetes.io/projected/d7906b48-2326-4074-a2a4-a62654c2b785-kube-api-access-gn4qt\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882459 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882523 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882576 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f811312-1c2a-4099-9cbe-efe1c5063b66-tmp\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882609 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-netd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882700 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-multus-certs\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882712 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882760 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2h77\" (UniqueName: \"kubernetes.io/projected/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-kube-api-access-z2h77\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.882919 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e2a6cc3b-391d-4626-908a-5466994e21d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883041 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883102 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883145 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883058 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883252 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-grpc-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883295 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e2a6cc3b-391d-4626-908a-5466994e21d5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883376 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-slash\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883440 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gd4\" (UniqueName: \"kubernetes.io/projected/82127ba1-4bef-43cd-8550-719a4402fcab-kube-api-access-54gd4\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.883555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883487 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1cd5923-8633-4b24-8310-e06ac0a2de68-metrics-tls\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883663 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/216e2c3d-4562-43b7-a06b-30743fa0cf0c-data-volume\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883713 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883671 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883722 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-root\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883793 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-etc-selinux\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883830 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/216e2c3d-4562-43b7-a06b-30743fa0cf0c-data-volume\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883852 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883865 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h6qwn\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883876 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/269d1132-4f1d-4b95-b545-2b2659d245a7-tmp\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883918 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883925 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883968 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883670 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883998 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883976 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883992 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883887 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.884064 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.884073 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-modprobe-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883993 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.883744 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8kj45\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.884157 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.884141 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-bin\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.884814 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cd5923-8633-4b24-8310-e06ac0a2de68-config-volume\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.884911 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8bc620-2f0c-4060-b76e-2b2365e3621f-hosts-file\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.884971 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:01:46.885098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885014 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-socket-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885150 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885379 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh65\" (UniqueName: \"kubernetes.io/projected/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-api-access-5bh65\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885402 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885434 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjpt\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885442 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885492 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82127ba1-4bef-43cd-8550-719a4402fcab-ovn-node-metrics-cert\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885547 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrg4j\" (UniqueName: \"kubernetes.io/projected/0bbee04d-3f06-4ce5-8020-4d52b9715887-kube-api-access-vrg4j\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885745 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7906b48-2326-4074-a2a4-a62654c2b785-host-slash\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885801 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-metrics-client-ca\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885979 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885985 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz24\" (UniqueName: \"kubernetes.io/projected/630626e4-d3eb-4e14-8c9d-9363718e5049-kube-api-access-4vz24\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:46.886433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.886379 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.887011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.886504 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:01:46.887011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.886576 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" Oct 30 18:01:46.887011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.886474 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-systemd-units\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.887011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.886841 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-multus\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.887011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.886896 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8nm\" (UniqueName: \"kubernetes.io/projected/9e62ca62-c183-4859-8d52-a52aa8182c73-kube-api-access-ch8nm\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:46.887011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.886988 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc620-2f0c-4060-b76e-2b2365e3621f-tmp-dir\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:01:46.887305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887037 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.887305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.885492 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cd5923-8633-4b24-8310-e06ac0a2de68-config-volume\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:46.887305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887079 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-config\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.887305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887103 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/269d1132-4f1d-4b95-b545-2b2659d245a7-tmp\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.887305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887259 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/82127ba1-4bef-43cd-8550-719a4402fcab-ovn-node-metrics-cert\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.887514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887339 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-netns\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.887514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887390 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntf6\" (UniqueName: \"kubernetes.io/projected/3275088d-42df-41bd-8b42-b739a353962d-kube-api-access-vntf6\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.887651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887614 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:46.888013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887768 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-env-overrides\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.888013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.887839 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" Oct 30 18:01:46.888212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.888133 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-ovnkube-config\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.888212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.888173 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.888467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.888422 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.888648 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.888492 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.888648 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.888565 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:46.888786 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.888728 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:01:46.888904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.888865 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.889051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.888953 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269d1132-4f1d-4b95-b545-2b2659d245a7-metrics-client-ca\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.889221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889194 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.889221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889211 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:01:46.889476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889249 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8bc620-2f0c-4060-b76e-2b2365e3621f-tmp-dir\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:01:46.889476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889362 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.889476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889408 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.889476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889465 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" Oct 30 18:01:46.889745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889675 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/82127ba1-4bef-43cd-8550-719a4402fcab-env-overrides\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.889826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889760 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-run\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.889904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889826 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-ovn\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.890025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.889899 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.890085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.890039 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-kubelet\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.890139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.890089 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/216e2c3d-4562-43b7-a06b-30743fa0cf0c-crio-socket\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.890214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.890162 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:01:46.890439 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.890406 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.890554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.890507 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-system-cni-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.890612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.890577 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-system-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.890983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.890925 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:01:46.891128 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.890975 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.891346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891320 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:46.891510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891485 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.891583 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891551 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-serviceca\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:01:46.891658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891596 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chz5j\" (UniqueName: \"kubernetes.io/projected/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-kube-api-access-chz5j\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.891658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891641 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cd5923-8633-4b24-8310-e06ac0a2de68-tmp-dir\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:46.891784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891693 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" Oct 30 18:01:46.891784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891708 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 18:01:46.891784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891722 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnt9h\" (UniqueName: \"kubernetes.io/projected/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-api-access-dnt9h\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.891995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891803 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 18:01:46.892214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.891824 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vph5m\" (UniqueName: \"kubernetes.io/projected/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kube-api-access-vph5m\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:46.892349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.892320 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.892425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.892356 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-binary-copy\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.892489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.892442 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.892545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.892503 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.892608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.892587 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f9bu29dqg7ac4\"" Oct 30 18:01:46.893085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893048 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-systemd\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.893209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893112 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-kubelet-config\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:46.893209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893166 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.893334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893230 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhg7\" (UniqueName: \"kubernetes.io/projected/b1cd5923-8633-4b24-8310-e06ac0a2de68-kube-api-access-nlhg7\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:46.893334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893278 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7906b48-2326-4074-a2a4-a62654c2b785-iptables-alerter-script\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:01:46.893334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893323 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-sys\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.893642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893564 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.893752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893723 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:01:46.893818 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893772 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:01:46.894012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.893962 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d4acf9b8-e56f-4958-9742-7303e972ae20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.894122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.894068 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-host\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:01:46.894122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.894101 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-grpc-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.894122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.894114 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-kubelet\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.894258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.894112 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7906b48-2326-4074-a2a4-a62654c2b785-iptables-alerter-script\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:01:46.894308 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.894277 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1cd5923-8633-4b24-8310-e06ac0a2de68-tmp-dir\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:46.894358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.894326 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:01:46.894798 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.894775 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" Oct 30 18:01:46.894984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.894921 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.895089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895036 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.895183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895162 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx7z4\" (UniqueName: \"kubernetes.io/projected/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-kube-api-access-dx7z4\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:01:46.895324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895265 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-serviceca\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:01:46.895399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895376 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gzppf\"" Oct 30 18:01:46.895468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895443 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hhk6k\"" Oct 30 18:01:46.895533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895079 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 18:01:46.895594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895523 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:46.895594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895572 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 18:01:46.895594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895580 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d05019b-efc6-444a-986d-6f63420ba3ff-konnectivity-ca\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:01:46.895738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895647 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/216e2c3d-4562-43b7-a06b-30743fa0cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.895738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895699 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-device-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:46.895912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895777 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.896003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895914 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1cd5923-8633-4b24-8310-e06ac0a2de68-metrics-tls\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:46.896003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.895924 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" Oct 30 18:01:46.896131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896053 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-lib-modules\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.896131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896116 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.896237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896161 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-conf-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.896292 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896261 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-tls\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.896292 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896273 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:01:46.896405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896274 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.896405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896304 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-sys\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.896405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896357 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7kw\" (UniqueName: \"kubernetes.io/projected/5f811312-1c2a-4099-9cbe-efe1c5063b66-kube-api-access-qw7kw\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.896405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896373 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bbee04d-3f06-4ce5-8020-4d52b9715887-metrics-certs\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:46.896549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896424 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-etc-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.896602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896558 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.896805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896751 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cni-binary-copy\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.896994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896833 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-sys-fs\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:46.896994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896836 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d05019b-efc6-444a-986d-6f63420ba3ff-konnectivity-ca\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:01:46.896994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.896914 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-tls\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.897156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.897001 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-netns\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.897156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.897108 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqpm\" (UniqueName: \"kubernetes.io/projected/d4acf9b8-e56f-4958-9742-7303e972ae20-kube-api-access-qqqpm\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.897271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.897246 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cni-binary-copy\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.897337 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.897304 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:46.897412 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.897391 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e62ca62-c183-4859-8d52-a52aa8182c73-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:46.897529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.897511 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d05019b-efc6-444a-986d-6f63420ba3ff-agent-certs\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:01:46.898132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898109 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.898271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898248 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.898361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898275 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:46.898361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898297 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-kubernetes\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.898466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898391 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.898466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898413 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-systemd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.898572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898460 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d05019b-efc6-444a-986d-6f63420ba3ff-agent-certs\") pod \"konnectivity-agent-6w92c\" (UID: \"5d05019b-efc6-444a-986d-6f63420ba3ff\") " pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:01:46.898572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898468 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.898572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898478 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18mfosc96e6jj\"" Oct 30 18:01:46.898572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898375 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898588 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-textfile\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898606 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898624 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898643 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898658 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q92vx\"" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898682 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-cnibin\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898694 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-accelerators-collector-config\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898708 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-textfile\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898798 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:46.898825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898814 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898822 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898849 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898946 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898964 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.898995 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899081 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899177 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899248 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3275088d-42df-41bd-8b42-b739a353962d-metrics-client-ca\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899255 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-bin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899313 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-registration-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899364 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/216e2c3d-4562-43b7-a06b-30743fa0cf0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899394 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.899394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899400 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/630626e4-d3eb-4e14-8c9d-9363718e5049-cert\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:46.899828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899469 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.899828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899552 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.899828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899593 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.899828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899597 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9cf883-7d59-46f0-b457-f84c7c678251-original-pull-secret\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:46.899828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899623 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.899828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899696 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-dbus\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:46.899828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899755 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxttb\" (UniqueName: \"kubernetes.io/projected/269d1132-4f1d-4b95-b545-2b2659d245a7-kube-api-access-gxttb\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.899828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899791 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/630626e4-d3eb-4e14-8c9d-9363718e5049-cert\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:46.900178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899799 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.900178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899891 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.900178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.899987 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.900178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.900144 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/269d1132-4f1d-4b95-b545-2b2659d245a7-secret-thanos-querier-tls\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.900504 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.900469 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.900609 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.900563 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.900903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.900871 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:01:46.900960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.900886 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.900993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.900977 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/269d1132-4f1d-4b95-b545-2b2659d245a7-metrics-client-ca\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.901025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.900991 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-etc-kubernetes\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.901025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.900990 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.901100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901023 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.901100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901084 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4qt\" (UniqueName: \"kubernetes.io/projected/d7906b48-2326-4074-a2a4-a62654c2b785-kube-api-access-gn4qt\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:01:46.901179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901162 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cnibin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.901220 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901199 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-socket-dir-parent\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.901220 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901201 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.901290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901248 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-hostroot\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.901290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901254 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:46.901362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901350 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krrq\" (UniqueName: \"kubernetes.io/projected/3b8bc620-2f0c-4060-b76e-2b2365e3621f-kube-api-access-6krrq\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:01:46.901398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901383 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.901433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901412 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.901487 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901441 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysconfig\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.901487 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901466 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-daemon-config\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.901733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901711 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.901905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901883 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-daemon-config\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.901998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.901883 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.906970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.906862 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.908018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.907953 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:46.908602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.908556 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9e62ca62-c183-4859-8d52-a52aa8182c73-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:46.908842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.908787 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gd4\" (UniqueName: \"kubernetes.io/projected/82127ba1-4bef-43cd-8550-719a4402fcab-kube-api-access-54gd4\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:46.909371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.909138 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2h77\" (UniqueName: \"kubernetes.io/projected/bf49c75c-fb11-4c2d-ac07-b41d85f8f106-kube-api-access-z2h77\") pod \"openshift-state-metrics-7bd7489957-pjv29\" (UID: \"bf49c75c-fb11-4c2d-ac07-b41d85f8f106\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pjv29" Oct 30 18:01:46.909517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.909388 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh65\" (UniqueName: \"kubernetes.io/projected/e2a6cc3b-391d-4626-908a-5466994e21d5-kube-api-access-5bh65\") pod \"kube-state-metrics-5dc8d7d89b-brcll\" (UID: \"e2a6cc3b-391d-4626-908a-5466994e21d5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" Oct 30 18:01:46.909725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.909571 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz24\" (UniqueName: \"kubernetes.io/projected/630626e4-d3eb-4e14-8c9d-9363718e5049-kube-api-access-4vz24\") pod \"ingress-canary-9l55k\" (UID: \"630626e4-d3eb-4e14-8c9d-9363718e5049\") " pod="openshift-ingress-canary/ingress-canary-9l55k" Oct 30 18:01:46.910359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.910296 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrg4j\" (UniqueName: \"kubernetes.io/projected/0bbee04d-3f06-4ce5-8020-4d52b9715887-kube-api-access-vrg4j\") pod \"network-metrics-daemon-hchmm\" (UID: \"0bbee04d-3f06-4ce5-8020-4d52b9715887\") " pod="openshift-multus/network-metrics-daemon-hchmm" Oct 30 18:01:46.911078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.911034 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjpt\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.911243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.911074 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chz5j\" (UniqueName: \"kubernetes.io/projected/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-kube-api-access-chz5j\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:46.911243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.911120 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:46.911243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.911072 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmv5\" (UniqueName: \"kubernetes.io/projected/7e4312f3-14a2-49b4-a6f1-4484d94a2217-kube-api-access-8pmv5\") pod \"network-check-target-xpbwl\" (UID: \"7e4312f3-14a2-49b4-a6f1-4484d94a2217\") " pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:46.911243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.911166 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntf6\" (UniqueName: \"kubernetes.io/projected/3275088d-42df-41bd-8b42-b739a353962d-kube-api-access-vntf6\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:46.911824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.911780 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8nm\" (UniqueName: \"kubernetes.io/projected/9e62ca62-c183-4859-8d52-a52aa8182c73-kube-api-access-ch8nm\") pod \"monitoring-plugin-77f4f76594-f87jc\" (UID: \"9e62ca62-c183-4859-8d52-a52aa8182c73\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:46.912653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.912586 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnt9h\" (UniqueName: \"kubernetes.io/projected/216e2c3d-4562-43b7-a06b-30743fa0cf0c-kube-api-access-dnt9h\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:46.913245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.913212 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vph5m\" (UniqueName: \"kubernetes.io/projected/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kube-api-access-vph5m\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:46.913614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.913582 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhg7\" (UniqueName: \"kubernetes.io/projected/b1cd5923-8633-4b24-8310-e06ac0a2de68-kube-api-access-nlhg7\") pod \"dns-default-6mcgq\" (UID: \"b1cd5923-8633-4b24-8310-e06ac0a2de68\") " pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:46.913843 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.913813 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx7z4\" (UniqueName: \"kubernetes.io/projected/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-kube-api-access-dx7z4\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:01:46.914060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.914034 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqpm\" (UniqueName: \"kubernetes.io/projected/d4acf9b8-e56f-4958-9742-7303e972ae20-kube-api-access-qqqpm\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:46.914134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.914108 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7kw\" (UniqueName: \"kubernetes.io/projected/5f811312-1c2a-4099-9cbe-efe1c5063b66-kube-api-access-qw7kw\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:46.914184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.914133 5624 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.914341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.914320 5624 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.916504 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.916446 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxttb\" (UniqueName: \"kubernetes.io/projected/269d1132-4f1d-4b95-b545-2b2659d245a7-kube-api-access-gxttb\") pod \"thanos-querier-74f988c6b9-jw4rr\" (UID: \"269d1132-4f1d-4b95-b545-2b2659d245a7\") " pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:46.917343 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.917298 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krrq\" (UniqueName: \"kubernetes.io/projected/3b8bc620-2f0c-4060-b76e-2b2365e3621f-kube-api-access-6krrq\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:01:46.937976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.937844 5624 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:01:46.937976 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:46.937977 5624 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-47.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" Oct 30 18:01:46.939563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:46.939513 5624 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:01:46.939864 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:46.939627 5624 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-47.ec2.internal" Oct 30 18:01:47.002724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002540 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-bin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.002724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002622 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-registration-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.002724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002667 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-dbus\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:47.002724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002675 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-bin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.002724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002701 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-etc-kubernetes\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.002724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002730 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cnibin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002765 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-socket-dir-parent\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002791 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-hostroot\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002834 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysconfig\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002851 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-cnibin\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002867 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-hostroot\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002783 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-registration-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002872 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-var-lib-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002875 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-dbus\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002916 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-var-lib-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.002961 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-node-log\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003003 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-etc-kubernetes\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003036 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-var-lib-kubelet\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003085 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-socket-dir-parent\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003144 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-node-log\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003181 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-log-socket\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003196 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-var-lib-kubelet\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003222 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-k8s-cni-cncf-io\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003332 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-log-socket\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003424 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysconfig\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.003479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003284 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-k8s-cni-cncf-io\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003487 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003556 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-host\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003637 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-host\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003676 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003704 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003726 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-wtmp\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003757 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-conf\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003763 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003796 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003827 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-netd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003854 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-multus-certs\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003895 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-slash\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003951 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-root\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003972 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-node-exporter-wtmp\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003982 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-sysctl-conf\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003987 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-netd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.003983 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-etc-selinux\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004018 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-slash\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004023 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-root\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004039 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004061 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-multus-certs\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004081 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-etc-selinux\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004110 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-modprobe-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004159 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-bin\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004208 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-cni-bin\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004207 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8bc620-2f0c-4060-b76e-2b2365e3621f-hosts-file\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004249 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-socket-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004262 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3b8bc620-2f0c-4060-b76e-2b2365e3621f-hosts-file\") pod \"node-resolver-n97j8\" (UID: \"3b8bc620-2f0c-4060-b76e-2b2365e3621f\") " pod="openshift-dns/node-resolver-n97j8" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004278 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7906b48-2326-4074-a2a4-a62654c2b785-host-slash\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004304 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7906b48-2326-4074-a2a4-a62654c2b785-host-slash\") pod \"iptables-alerter-fl2ts\" (UID: \"d7906b48-2326-4074-a2a4-a62654c2b785\") " pod="openshift-network-operator/iptables-alerter-fl2ts" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004280 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-modprobe-d\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004316 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-socket-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004303 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-systemd-units\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004343 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-systemd-units\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004349 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-multus\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004375 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-cni-multus\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004382 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-netns\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004406 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004411 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-run-netns\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004428 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004460 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-run\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004489 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-ovn\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004509 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-kubelet\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004517 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004528 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004539 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/216e2c3d-4562-43b7-a06b-30743fa0cf0c-crio-socket\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004548 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-run\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.004523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004574 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-system-cni-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004585 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-host-var-lib-kubelet\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004606 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-ovn\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004609 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-system-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004649 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-system-cni-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004665 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-system-cni-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004678 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-systemd\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004609 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/216e2c3d-4562-43b7-a06b-30743fa0cf0c-crio-socket\") pod \"insights-runtime-extractor-x6tdb\" (UID: \"216e2c3d-4562-43b7-a06b-30743fa0cf0c\") " pod="openshift-insights/insights-runtime-extractor-x6tdb" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004703 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-kubelet-config\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004734 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-sys\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004743 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-systemd\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004757 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-host\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004779 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9cf883-7d59-46f0-b457-f84c7c678251-kubelet-config\") pod \"global-pull-secret-syncer-qg2lp\" (UID: \"8e9cf883-7d59-46f0-b457-f84c7c678251\") " pod="kube-system/global-pull-secret-syncer-qg2lp" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004782 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3275088d-42df-41bd-8b42-b739a353962d-sys\") pod \"node-exporter-7bjq6\" (UID: \"3275088d-42df-41bd-8b42-b739a353962d\") " pod="openshift-monitoring/node-exporter-7bjq6" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004825 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-kubelet\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004786 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-kubelet\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004904 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-device-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004808 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2-host\") pod \"node-ca-l6vjs\" (UID: \"c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2\") " pod="openshift-image-registry/node-ca-l6vjs" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.004961 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-lib-modules\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005007 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-conf-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005015 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-device-dir\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005051 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-sys\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005120 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-lib-modules\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005138 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-etc-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005102 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-sys\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005098 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/39764b0f-c4d1-4758-b8c2-9fb4d001ed8b-multus-conf-dir\") pod \"multus-sq5cb\" (UID: \"39764b0f-c4d1-4758-b8c2-9fb4d001ed8b\") " pod="openshift-multus/multus-sq5cb" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005167 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005216 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005218 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-etc-openvswitch\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005223 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-sys-fs\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005280 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ca1d631-c3c1-4e4f-910c-76a1310a327c-sys-fs\") pod \"aws-ebs-csi-driver-node-55lt6\" (UID: \"8ca1d631-c3c1-4e4f-910c-76a1310a327c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005323 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-netns\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005363 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-kubernetes\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005383 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-systemd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005404 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005426 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-host-run-netns\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005437 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-cnibin\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005456 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/82127ba1-4bef-43cd-8550-719a4402fcab-run-systemd\") pod \"ovnkube-node-g74ls\" (UID: \"82127ba1-4bef-43cd-8550-719a4402fcab\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005457 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f811312-1c2a-4099-9cbe-efe1c5063b66-etc-kubernetes\") pod \"tuned-k8mxn\" (UID: \"5f811312-1c2a-4099-9cbe-efe1c5063b66\") " pod="openshift-cluster-node-tuning-operator/tuned-k8mxn" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005488 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-cnibin\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:47.006962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.005487 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d4acf9b8-e56f-4958-9742-7303e972ae20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gdxmg\" (UID: \"d4acf9b8-e56f-4958-9742-7303e972ae20\") " pod="openshift-multus/multus-additional-cni-plugins-gdxmg" Oct 30 18:01:47.012207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.012144 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:47.148908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.148843 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" Oct 30 18:01:47.164492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:47.164436 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:01:49.397157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:49.397106 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6mcgq" Oct 30 18:01:49.929620 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:01:49.929541 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:01:50.592309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:50.592236 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:50.692274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:50.692206 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:51.378347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:51.378280 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" Oct 30 18:01:53.913115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:53.912975 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:54.134628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:54.134559 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6w92c" Oct 30 18:01:56.369646 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:56.369562 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:01:56.567354 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:56.567282 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:01:56.567681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:01:56.567646 5624 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:02:05.887295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:05.887220 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:06.023211 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:06.023130 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:08.524876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:08.524826 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" Oct 30 18:02:11.852278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:11.852197 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:02:11.853393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:11.853284 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="prometheus" containerID="cri-o://de0a1c6b124af964f263dcb8709d0187ff15a67ef8ebcd2fa89a2d371318f8cd" gracePeriod=600 Oct 30 18:02:11.853725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:11.853690 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="thanos-sidecar" containerID="cri-o://970d5eb05250e54cd9fad61f9bc850db3b2dfc29300b17cbe9fb5fd2bd924e36" gracePeriod=600 Oct 30 18:02:11.853883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:11.853694 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy-web" containerID="cri-o://d3eeeea5025d5952b94ab67d83d3ad002c7d433fe276419aa3bf7e2cae46658c" gracePeriod=600 Oct 30 18:02:11.853883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:11.853744 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2e9a900703b978498f1d64f21da42dee80524b69c8cd216558c8c14816485d12" gracePeriod=600 Oct 30 18:02:11.853883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:11.853819 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="config-reloader" containerID="cri-o://67879e3838023e126dd9d1dcc52fdfc864bcbef6d23dcf11c8e9b35480ee4120" gracePeriod=600 Oct 30 18:02:11.854125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:11.853767 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy" containerID="cri-o://b034530536b45df40c271a308eb4c45a2178b363d6965baf087b534fdb50054a" gracePeriod=600 Oct 30 18:02:12.055977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.055893 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerID="2e9a900703b978498f1d64f21da42dee80524b69c8cd216558c8c14816485d12" exitCode=0 Oct 30 18:02:12.055977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.055963 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerID="b034530536b45df40c271a308eb4c45a2178b363d6965baf087b534fdb50054a" exitCode=0 Oct 30 18:02:12.055977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.055973 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerID="d3eeeea5025d5952b94ab67d83d3ad002c7d433fe276419aa3bf7e2cae46658c" exitCode=0 Oct 30 18:02:12.055977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.055982 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerID="970d5eb05250e54cd9fad61f9bc850db3b2dfc29300b17cbe9fb5fd2bd924e36" exitCode=0 Oct 30 18:02:12.055977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.055995 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerID="67879e3838023e126dd9d1dcc52fdfc864bcbef6d23dcf11c8e9b35480ee4120" exitCode=0 Oct 30 18:02:12.055977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.056005 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerID="de0a1c6b124af964f263dcb8709d0187ff15a67ef8ebcd2fa89a2d371318f8cd" exitCode=0 Oct 30 18:02:12.056559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.055986 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerDied","Data":"2e9a900703b978498f1d64f21da42dee80524b69c8cd216558c8c14816485d12"} Oct 30 18:02:12.056559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.056146 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerDied","Data":"b034530536b45df40c271a308eb4c45a2178b363d6965baf087b534fdb50054a"} Oct 30 18:02:12.056559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.056183 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerDied","Data":"d3eeeea5025d5952b94ab67d83d3ad002c7d433fe276419aa3bf7e2cae46658c"} Oct 30 18:02:12.056559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.056216 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerDied","Data":"970d5eb05250e54cd9fad61f9bc850db3b2dfc29300b17cbe9fb5fd2bd924e36"} Oct 30 18:02:12.056559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.056249 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerDied","Data":"67879e3838023e126dd9d1dcc52fdfc864bcbef6d23dcf11c8e9b35480ee4120"} Oct 30 18:02:12.056559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.056287 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerDied","Data":"de0a1c6b124af964f263dcb8709d0187ff15a67ef8ebcd2fa89a2d371318f8cd"} Oct 30 18:02:12.132603 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:02:12.132500 5624 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="de0a1c6b124af964f263dcb8709d0187ff15a67ef8ebcd2fa89a2d371318f8cd" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 18:02:12.132976 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:02:12.132671 5624 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="de0a1c6b124af964f263dcb8709d0187ff15a67ef8ebcd2fa89a2d371318f8cd" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 18:02:12.132976 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:02:12.132713 5624 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="de0a1c6b124af964f263dcb8709d0187ff15a67ef8ebcd2fa89a2d371318f8cd" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 18:02:12.132976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.132740 5624 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 30 18:02:12.185797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.185727 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:12.283653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.283595 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.283653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.283673 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.283715 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.283757 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.283782 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.283815 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.283896 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.283985 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284028 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284087 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284128 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284179 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284213 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284269 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284313 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284359 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284399 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284446 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284534 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.284608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.284576 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt\") pod \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\" (UID: \"cc1afeda-d283-47f1-978a-43aeeae7d6cb\") " Oct 30 18:02:12.286447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.285745 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:12.286447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.285900 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:12.286789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.286543 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:12.287151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.287053 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:12.287668 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.287560 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:12.289601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.289504 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:12.290618 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.290357 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.290618 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.290441 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:12.291183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.291058 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.291837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.291339 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:12.291837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.291366 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.291837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.291459 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:12.291837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.291611 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.292144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.292034 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "kube-api-access-vbjpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:12.292436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.292400 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.294074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.293997 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.294588 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.294529 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.294726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.294597 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:12.294726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.294645 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.312300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.312212 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config" pod "cc1afeda-d283-47f1-978a-43aeeae7d6cb" (UID: "cc1afeda-d283-47f1-978a-43aeeae7d6cb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.385802 5624 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-kube-rbac-proxy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.385868 5624 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-metrics-client-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.385881 5624 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.385896 5624 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.385908 5624 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-metrics-client-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.385919 5624 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.385930 5624 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.385995 5624 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-grpc-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386009 5624 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config-out\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386059 5624 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-tls-assets\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386071 5624 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386082 5624 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-db\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386093 5624 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386105 5624 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386115 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbjpt\" (UniqueName: \"kubernetes.io/projected/cc1afeda-d283-47f1-978a-43aeeae7d6cb-kube-api-access-vbjpt\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386125 5624 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386135 5624 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386144 5624 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc1afeda-d283-47f1-978a-43aeeae7d6cb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386153 5624 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-config\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:12.386551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:12.386162 5624 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc1afeda-d283-47f1-978a-43aeeae7d6cb-web-config\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:02:13.068878 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.068804 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc1afeda-d283-47f1-978a-43aeeae7d6cb","Type":"ContainerDied","Data":"c627932e68de4b280509b3b4a67ba6ec8dba4dc843e0bfb4d1421cc4b8c8124a"} Oct 30 18:02:13.068878 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.068878 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.069613 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.069024 5624 scope.go:122] "RemoveContainer" containerID="2e9a900703b978498f1d64f21da42dee80524b69c8cd216558c8c14816485d12" Oct 30 18:02:13.084373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.084325 5624 scope.go:122] "RemoveContainer" containerID="b034530536b45df40c271a308eb4c45a2178b363d6965baf087b534fdb50054a" Oct 30 18:02:13.098478 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.098439 5624 scope.go:122] "RemoveContainer" containerID="d3eeeea5025d5952b94ab67d83d3ad002c7d433fe276419aa3bf7e2cae46658c" Oct 30 18:02:13.111655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.111616 5624 scope.go:122] "RemoveContainer" containerID="970d5eb05250e54cd9fad61f9bc850db3b2dfc29300b17cbe9fb5fd2bd924e36" Oct 30 18:02:13.125567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.125530 5624 scope.go:122] "RemoveContainer" containerID="67879e3838023e126dd9d1dcc52fdfc864bcbef6d23dcf11c8e9b35480ee4120" Oct 30 18:02:13.137318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.137186 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:02:13.140536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.140497 5624 scope.go:122] "RemoveContainer" containerID="de0a1c6b124af964f263dcb8709d0187ff15a67ef8ebcd2fa89a2d371318f8cd" Oct 30 18:02:13.148498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.148445 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:02:13.156303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.156269 5624 scope.go:122] "RemoveContainer" containerID="eeaa04d40d7f51db91196146c7868ea79da05945fb9a4f55784220ab9404ea8c" Oct 30 18:02:13.170634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.170558 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:02:13.171255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171219 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="prometheus" Oct 30 18:02:13.171255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171259 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="prometheus" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171284 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy-web" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171295 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy-web" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171322 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="config-reloader" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171331 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="config-reloader" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171341 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="init-config-reloader" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171351 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="init-config-reloader" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171370 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="thanos-sidecar" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171377 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="thanos-sidecar" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171392 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy-thanos" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171400 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy-thanos" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171423 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy" Oct 30 18:02:13.171502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171431 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy" Oct 30 18:02:13.172100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171540 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="thanos-sidecar" Oct 30 18:02:13.172100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171560 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy-thanos" Oct 30 18:02:13.172100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171572 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy" Oct 30 18:02:13.172100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171618 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="prometheus" Oct 30 18:02:13.172100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171629 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="kube-rbac-proxy-web" Oct 30 18:02:13.172100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171640 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="config-reloader" Oct 30 18:02:13.172100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.171649 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" containerName="init-config-reloader" Oct 30 18:02:13.179107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.179042 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.197153 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197107 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:02:13.197153 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197120 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:02:13.197635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197300 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:02:13.197635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197353 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:02:13.197635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197396 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:02:13.197635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197604 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gzppf\"" Oct 30 18:02:13.197635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197600 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:02:13.197911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197606 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:02:13.197911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197667 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:02:13.197911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197667 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18mfosc96e6jj\"" Oct 30 18:02:13.197911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.197703 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:02:13.198528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.198432 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:02:13.198528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.198443 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:02:13.198870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.198641 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:02:13.198870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.198712 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:02:13.211169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.211109 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:02:13.211465 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.211179 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:02:13.212105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.212056 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:02:13.296862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.296778 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.296862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.296867 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.296962 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297022 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-config\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297147 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297180 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-web-config\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297214 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297283 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297362 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297406 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297448 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297536 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297594 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297740 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297717 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ade95085-ca7b-4e3b-9883-9dca382e1d69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297763 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ade95085-ca7b-4e3b-9883-9dca382e1d69-config-out\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297790 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297865 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.297972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297918 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.298019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.297986 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.298124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.298102 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmrsv\" (UniqueName: \"kubernetes.io/projected/ade95085-ca7b-4e3b-9883-9dca382e1d69-kube-api-access-gmrsv\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.399549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399351 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.399549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399437 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.399549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399470 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.399549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399499 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.399549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399540 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399575 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ade95085-ca7b-4e3b-9883-9dca382e1d69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399604 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ade95085-ca7b-4e3b-9883-9dca382e1d69-config-out\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399628 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399673 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399706 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399813 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399853 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmrsv\" (UniqueName: \"kubernetes.io/projected/ade95085-ca7b-4e3b-9883-9dca382e1d69-kube-api-access-gmrsv\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399916 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.399995 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400035 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400100 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-config\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400138 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400167 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-web-config\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400210 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400237 5624 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:02:13.400790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400563 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.400790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400674 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.401135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400916 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.401135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.401124 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.401251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.400248 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.401481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.401403 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.402926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.402402 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.402926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.402409 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.405226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.404643 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.405226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.404705 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.405226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.404801 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.405226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.404841 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.405226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.405063 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.407012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.406828 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ade95085-ca7b-4e3b-9883-9dca382e1d69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.407308 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.407058 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-web-config\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.407740 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.407701 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-config\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.408410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.408359 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.408604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.408357 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ade95085-ca7b-4e3b-9883-9dca382e1d69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.408698 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.408610 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ade95085-ca7b-4e3b-9883-9dca382e1d69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.408698 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.408659 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ade95085-ca7b-4e3b-9883-9dca382e1d69-config-out\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.422165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.422112 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmrsv\" (UniqueName: \"kubernetes.io/projected/ade95085-ca7b-4e3b-9883-9dca382e1d69-kube-api-access-gmrsv\") pod \"prometheus-k8s-0\" (UID: \"ade95085-ca7b-4e3b-9883-9dca382e1d69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.516923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.516827 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:13.796446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.796280 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1afeda-d283-47f1-978a-43aeeae7d6cb" path="/var/lib/kubelet/pods/cc1afeda-d283-47f1-978a-43aeeae7d6cb/volumes" Oct 30 18:02:13.879772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:13.879702 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:02:13.886548 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:02:13.886477 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade95085_ca7b_4e3b_9883_9dca382e1d69.slice/crio-d317512721cb9aac4dda9e7ea3e24335f6cc09ce31d04022cc1225fc691c1430 WatchSource:0}: Error finding container d317512721cb9aac4dda9e7ea3e24335f6cc09ce31d04022cc1225fc691c1430: Status 404 returned error can't find the container with id d317512721cb9aac4dda9e7ea3e24335f6cc09ce31d04022cc1225fc691c1430 Oct 30 18:02:14.079563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:14.079406 5624 generic.go:356] "Generic (PLEG): container finished" podID="ade95085-ca7b-4e3b-9883-9dca382e1d69" containerID="64d523b71d4e373d063bdb248e86fa6c3bec30af12f99621123323345013bd9b" exitCode=0 Oct 30 18:02:14.079563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:14.079515 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ade95085-ca7b-4e3b-9883-9dca382e1d69","Type":"ContainerDied","Data":"64d523b71d4e373d063bdb248e86fa6c3bec30af12f99621123323345013bd9b"} Oct 30 18:02:14.080283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:14.079650 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ade95085-ca7b-4e3b-9883-9dca382e1d69","Type":"ContainerStarted","Data":"d317512721cb9aac4dda9e7ea3e24335f6cc09ce31d04022cc1225fc691c1430"} Oct 30 18:02:15.092213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.092149 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ade95085-ca7b-4e3b-9883-9dca382e1d69","Type":"ContainerStarted","Data":"5b4daf7185d1fe5bd98672033f4fecc8bd4e3e42e5cbdeca39e4544f7e742332"} Oct 30 18:02:15.093014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.092284 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ade95085-ca7b-4e3b-9883-9dca382e1d69","Type":"ContainerStarted","Data":"e1ca62c78c8c30919f30ea8f63c04965c2903677eb39aa12f4fbeb5582a0205f"} Oct 30 18:02:15.093014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.092337 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ade95085-ca7b-4e3b-9883-9dca382e1d69","Type":"ContainerStarted","Data":"ccd5ee3281e9a7584153a7ff64e439d3b225c840c3a857e1ac4876de03974681"} Oct 30 18:02:15.093014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.092380 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ade95085-ca7b-4e3b-9883-9dca382e1d69","Type":"ContainerStarted","Data":"7ea22a00c3474e338463996f654aa42b820a82d45b933feeeaae524cc57f4df4"} Oct 30 18:02:15.093014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.092424 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ade95085-ca7b-4e3b-9883-9dca382e1d69","Type":"ContainerStarted","Data":"bc35d55f9dda4a31c7f99ef10d8ca8e47479e7d44ebd2cccd6421976f798eaca"} Oct 30 18:02:15.093014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.092473 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ade95085-ca7b-4e3b-9883-9dca382e1d69","Type":"ContainerStarted","Data":"d325196cb10b688597ccfd6d409fa3580619ee8572eba140ab1490dc9501af46"} Oct 30 18:02:15.151155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.150837 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.12249098 podStartE2EDuration="2.150810949s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.319969ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:15.150138824 +0000 UTC m=+29.538666938" watchObservedRunningTime="2026-10-30 18:02:15.150810949 +0000 UTC m=+29.539339070" Oct 30 18:02:15.334712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.334629 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:15.444171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.443969 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:15.634112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.634052 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:15.736659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:15.736496 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:16.375188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:16.375134 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xpbwl" Oct 30 18:02:18.517737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:18.517643 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:19.930726 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:02:19.930658 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:02:25.806226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:25.806160 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:25.833104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:25.833049 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:26.178119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:26.178064 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:02:31.801681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:31.801576 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:38.583673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:38.583586 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:42.479654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:42.479562 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:42.579891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:42.579804 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:42.777556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:42.777365 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:42.903977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:42.903825 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:44.368860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:44.368767 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:44.460402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:44.460327 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:49.930997 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:02:49.930896 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:02:50.619426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:50.619323 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:50.716963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:50.716886 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:57.017116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:02:57.017040 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:19.928929 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:03:19.928843 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:03:38.709691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:03:38.709483 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:49.929634 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:03:49.929559 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:04:19.934042 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:04:19.933965 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:04:20.045137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:20.045064 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:21.398807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:21.398725 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:21.710535 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:21.710260 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:21.871858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:21.871801 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:22.607784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.607700 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8"] Oct 30 18:04:22.613000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.612910 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.620789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.620717 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.621132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.620862 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2cch\" (UniqueName: \"kubernetes.io/projected/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-kube-api-access-h2cch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.621132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.620991 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.639172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.639106 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:04:22.639172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.639134 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:04:22.640691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.640639 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8"] Oct 30 18:04:22.710071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.710003 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-9cqwf\"" Oct 30 18:04:22.721772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.721701 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2cch\" (UniqueName: \"kubernetes.io/projected/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-kube-api-access-h2cch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.722131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.721860 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.722131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.721957 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.722577 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.722543 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.722619 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.722569 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.748029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.747961 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2cch\" (UniqueName: \"kubernetes.io/projected/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-kube-api-access-h2cch\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.781654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.781568 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl"] Oct 30 18:04:22.787068 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.786989 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.801146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.801095 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-mmzvc\"" Oct 30 18:04:22.805265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.805208 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl"] Oct 30 18:04:22.823502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.823409 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.823889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.823654 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/5d6bb2f7-7884-4bab-8bcb-65176afa9134-kube-api-access-9nps7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.823889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.823721 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.867802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.867635 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z"] Oct 30 18:04:22.872673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.872611 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:22.886685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.886624 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-7vzjj\"" Oct 30 18:04:22.891486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.891414 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z"] Oct 30 18:04:22.924587 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.924498 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:22.924587 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.924583 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.925049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.924752 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/5d6bb2f7-7884-4bab-8bcb-65176afa9134-kube-api-access-9nps7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.925049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.924817 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.925049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.924880 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxbl\" (UniqueName: \"kubernetes.io/projected/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-kube-api-access-ztxbl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:22.925184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.925072 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:22.925324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.925286 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.925376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.925324 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.937440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.937380 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:04:22.946524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.946459 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/5d6bb2f7-7884-4bab-8bcb-65176afa9134-kube-api-access-9nps7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:22.977608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.977511 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt"] Oct 30 18:04:22.985412 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:22.985337 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.001161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.001094 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt"] Oct 30 18:04:23.003254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.003207 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8s5bq\"" Oct 30 18:04:23.013429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.013352 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:23.028807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.028092 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:23.028807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.028199 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.028807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.028272 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.028807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.028377 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxbl\" (UniqueName: \"kubernetes.io/projected/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-kube-api-access-ztxbl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:23.028807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.028483 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:23.028807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.028528 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxghs\" (UniqueName: \"kubernetes.io/projected/6606b549-4d64-49d2-9b98-cfce59198eb6-kube-api-access-sxghs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.029490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.029368 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:23.030705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.030665 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:23.061438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.058441 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxbl\" (UniqueName: \"kubernetes.io/projected/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-kube-api-access-ztxbl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:23.112599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.112503 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:04:23.129598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.129517 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxghs\" (UniqueName: \"kubernetes.io/projected/6606b549-4d64-49d2-9b98-cfce59198eb6-kube-api-access-sxghs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.129922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.129647 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.129922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.129714 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.130516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.130335 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.130516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.130345 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.160385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.160316 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxghs\" (UniqueName: \"kubernetes.io/projected/6606b549-4d64-49d2-9b98-cfce59198eb6-kube-api-access-sxghs\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.177560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.177495 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv"] Oct 30 18:04:23.184413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.184350 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.197610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.197550 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:04:23.201984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.200861 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-dd5x6\"" Oct 30 18:04:23.209125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.208882 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv"] Oct 30 18:04:23.231701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.231626 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.232020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.231712 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pwg\" (UniqueName: \"kubernetes.io/projected/404bc46d-2200-4328-9e0a-e82974d16395-kube-api-access-s5pwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.232020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.231910 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.283480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.283306 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8"] Oct 30 18:04:23.293821 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:23.291430 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddc67f4_2b56_40b9_ac6b_7fae3444d1c3.slice/crio-492a45a2398fbbeb3319fa07bd3741700f7ef9d42166022e88dbc784af240223 WatchSource:0}: Error finding container 492a45a2398fbbeb3319fa07bd3741700f7ef9d42166022e88dbc784af240223: Status 404 returned error can't find the container with id 492a45a2398fbbeb3319fa07bd3741700f7ef9d42166022e88dbc784af240223 Oct 30 18:04:23.296019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.295291 5624 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:04:23.320336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.320257 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:04:23.333841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.333775 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.334205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.334015 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pwg\" (UniqueName: \"kubernetes.io/projected/404bc46d-2200-4328-9e0a-e82974d16395-kube-api-access-s5pwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.334205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.334126 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.334444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.334415 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.334691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.334663 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.355558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.355468 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pwg\" (UniqueName: \"kubernetes.io/projected/404bc46d-2200-4328-9e0a-e82974d16395-kube-api-access-s5pwg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.427470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.427400 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq"] Oct 30 18:04:23.435706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.435642 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.449349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.449144 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl"] Oct 30 18:04:23.450816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.450731 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-x59zf\"" Oct 30 18:04:23.453107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.453022 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq"] Oct 30 18:04:23.456775 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:23.456719 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6bb2f7_7884_4bab_8bcb_65176afa9134.slice/crio-365dd00898fe1a45c932a766c6a931f23369100be76473c333eca6833f7dd93d WatchSource:0}: Error finding container 365dd00898fe1a45c932a766c6a931f23369100be76473c333eca6833f7dd93d: Status 404 returned error can't find the container with id 365dd00898fe1a45c932a766c6a931f23369100be76473c333eca6833f7dd93d Oct 30 18:04:23.530051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.529980 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:04:23.537155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.537087 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.537342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.537257 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.537430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.537404 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtws7\" (UniqueName: \"kubernetes.io/projected/843037eb-d856-4011-ab94-438be80ccb61-kube-api-access-qtws7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.541813 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.541741 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z"] Oct 30 18:04:23.549886 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:23.549830 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc8ce9e_74f2_4dd4_91d2_2a025720567c.slice/crio-7b1267ae89f40c10666b4154cb9c840ca1d22e44670d4cc54e7306bf12269ca5 WatchSource:0}: Error finding container 7b1267ae89f40c10666b4154cb9c840ca1d22e44670d4cc54e7306bf12269ca5: Status 404 returned error can't find the container with id 7b1267ae89f40c10666b4154cb9c840ca1d22e44670d4cc54e7306bf12269ca5 Oct 30 18:04:23.639550 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.638681 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtws7\" (UniqueName: \"kubernetes.io/projected/843037eb-d856-4011-ab94-438be80ccb61-kube-api-access-qtws7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.640486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.640418 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.641259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.639741 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.641259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.640630 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.641409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.641315 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.650965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.650760 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt"] Oct 30 18:04:23.658406 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:23.658342 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6606b549_4d64_49d2_9b98_cfce59198eb6.slice/crio-70dfb7356a6eba04fbe51530181e7dd4604bd69fd0120e0b3f39c9cf4990a2e6 WatchSource:0}: Error finding container 70dfb7356a6eba04fbe51530181e7dd4604bd69fd0120e0b3f39c9cf4990a2e6: Status 404 returned error can't find the container with id 70dfb7356a6eba04fbe51530181e7dd4604bd69fd0120e0b3f39c9cf4990a2e6 Oct 30 18:04:23.659926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.659879 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtws7\" (UniqueName: \"kubernetes.io/projected/843037eb-d856-4011-ab94-438be80ccb61-kube-api-access-qtws7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.764384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.764318 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:04:23.767605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.767497 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" event={"ID":"5bc8ce9e-74f2-4dd4-91d2-2a025720567c","Type":"ContainerStarted","Data":"7b1267ae89f40c10666b4154cb9c840ca1d22e44670d4cc54e7306bf12269ca5"} Oct 30 18:04:23.771244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.771136 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" event={"ID":"5d6bb2f7-7884-4bab-8bcb-65176afa9134","Type":"ContainerStarted","Data":"365dd00898fe1a45c932a766c6a931f23369100be76473c333eca6833f7dd93d"} Oct 30 18:04:23.778665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.778509 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" event={"ID":"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3","Type":"ContainerStarted","Data":"492a45a2398fbbeb3319fa07bd3741700f7ef9d42166022e88dbc784af240223"} Oct 30 18:04:23.783451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.783374 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" event={"ID":"6606b549-4d64-49d2-9b98-cfce59198eb6","Type":"ContainerStarted","Data":"70dfb7356a6eba04fbe51530181e7dd4604bd69fd0120e0b3f39c9cf4990a2e6"} Oct 30 18:04:23.868413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:23.868363 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv"] Oct 30 18:04:23.898277 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:23.898207 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404bc46d_2200_4328_9e0a_e82974d16395.slice/crio-72905cb7b825b671860cef06f9679b917cbd1cacd006b5bb9e5aa1e110699d39 WatchSource:0}: Error finding container 72905cb7b825b671860cef06f9679b917cbd1cacd006b5bb9e5aa1e110699d39: Status 404 returned error can't find the container with id 72905cb7b825b671860cef06f9679b917cbd1cacd006b5bb9e5aa1e110699d39 Oct 30 18:04:24.116456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:24.116351 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq"] Oct 30 18:04:24.806247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:24.806041 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" event={"ID":"404bc46d-2200-4328-9e0a-e82974d16395","Type":"ContainerStarted","Data":"72905cb7b825b671860cef06f9679b917cbd1cacd006b5bb9e5aa1e110699d39"} Oct 30 18:04:24.811847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:24.811469 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" event={"ID":"843037eb-d856-4011-ab94-438be80ccb61","Type":"ContainerStarted","Data":"37446d1122f95d30503bad8fdb11ea962c6123d9bb18ae0fff99d547a7a264b0"} Oct 30 18:04:26.529695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.529508 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:26.649528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.648804 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:26.760462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.760385 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h"] Oct 30 18:04:26.771871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.771815 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:26.788716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.788525 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h"] Oct 30 18:04:26.799151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.798732 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-s8c27\"" Oct 30 18:04:26.887440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.887366 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:26.887836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.887572 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qsml\" (UniqueName: \"kubernetes.io/projected/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-kube-api-access-5qsml\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:26.887836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.887684 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:26.989534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.989455 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:26.989873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.989643 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qsml\" (UniqueName: \"kubernetes.io/projected/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-kube-api-access-5qsml\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:26.989873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.989725 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:26.990374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.990336 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:26.990676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:26.990652 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:27.014196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.013924 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qsml\" (UniqueName: \"kubernetes.io/projected/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-kube-api-access-5qsml\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:27.109398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.108471 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:04:27.356742 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.356653 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7"] Oct 30 18:04:27.365018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.364849 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.391987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.384887 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jf5s7\"" Oct 30 18:04:27.391987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.386508 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:04:27.391987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.386651 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:04:27.391987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.386784 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:04:27.391987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.390090 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7"] Oct 30 18:04:27.396122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.396045 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.396718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.396674 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446ts\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-kube-api-access-446ts\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.396896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.396850 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.397009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.396904 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.397009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.396975 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.397110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.397030 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.397110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.397075 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.397216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.397181 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.397265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.397218 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da4ab2-eb1e-458b-93a5-c063d30768de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.420106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.419890 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.497540 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn"] Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500063 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500129 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446ts\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-kube-api-access-446ts\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500316 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500361 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500406 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500447 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500486 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500557 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.500592 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da4ab2-eb1e-458b-93a5-c063d30768de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.502031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.501864 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da4ab2-eb1e-458b-93a5-c063d30768de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.504957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.504242 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.504957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.504522 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.505300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.504996 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.505745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.505712 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.508869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.508752 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.509277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.509024 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.511135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.509825 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.519734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.519642 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h"] Oct 30 18:04:27.530649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.530453 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-xm7rs\"" Oct 30 18:04:27.535575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.532790 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn"] Oct 30 18:04:27.535575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.534608 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.540922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.538912 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:27.540922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.540056 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-446ts\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-kube-api-access-446ts\") pod \"maas-default-gateway-openshift-default-77f556d647-b9qj7\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.544336 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:27.544136 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c4271e_b155_4c31_ba6d_e7d7c4d7a9c1.slice/crio-ced1e08f39449cf097cdd6f71d7b182679efa9561ab9376a8b8afc2bb85e2a7b WatchSource:0}: Error finding container ced1e08f39449cf097cdd6f71d7b182679efa9561ab9376a8b8afc2bb85e2a7b: Status 404 returned error can't find the container with id ced1e08f39449cf097cdd6f71d7b182679efa9561ab9376a8b8afc2bb85e2a7b Oct 30 18:04:27.614393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.614309 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.615837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.615012 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.615837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.615123 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.615837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.615179 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.615837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.615234 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.615837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.615269 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmhx\" (UniqueName: \"kubernetes.io/projected/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-kube-api-access-8mmhx\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.615837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.615311 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.615837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.615405 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.615837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.615450 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.635745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.635558 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:27.703711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.703153 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:27.716453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716385 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.716453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716463 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.716877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716542 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.716877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716570 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.716877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716607 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.716877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716639 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.716877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716674 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.716877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716709 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmhx\" (UniqueName: \"kubernetes.io/projected/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-kube-api-access-8mmhx\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.716877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.716747 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.719152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.717528 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.719152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.717925 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.719152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.718526 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.719152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.718834 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.719152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.719087 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.735395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.735334 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.739049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.738461 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.749973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.749713 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmhx\" (UniqueName: \"kubernetes.io/projected/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-kube-api-access-8mmhx\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.756362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.756210 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1fa476-56bf-4fd4-a676-5da6109ab2ed-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn\" (UID: \"6a1fa476-56bf-4fd4-a676-5da6109ab2ed\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.865002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.863292 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:27.885196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:27.885123 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" event={"ID":"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1","Type":"ContainerStarted","Data":"ced1e08f39449cf097cdd6f71d7b182679efa9561ab9376a8b8afc2bb85e2a7b"} Oct 30 18:04:28.125316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.124251 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7"] Oct 30 18:04:28.285208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.284373 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn"] Oct 30 18:04:28.291223 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:28.291137 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1fa476_56bf_4fd4_a676_5da6109ab2ed.slice/crio-ade5fff8161b21658a2cce6cb131201d657c16cdbd1a46a1961cac196778a677 WatchSource:0}: Error finding container ade5fff8161b21658a2cce6cb131201d657c16cdbd1a46a1961cac196778a677: Status 404 returned error can't find the container with id ade5fff8161b21658a2cce6cb131201d657c16cdbd1a46a1961cac196778a677 Oct 30 18:04:28.677663 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.677592 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5"] Oct 30 18:04:28.687375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.687301 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.717326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.717162 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5"] Oct 30 18:04:28.719237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.719179 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg"] Oct 30 18:04:28.730158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.729584 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.742837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.742625 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.742837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.742831 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.743246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.742908 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85b5v\" (UniqueName: \"kubernetes.io/projected/9026e924-adb6-4a37-9e30-3ea721461bcd-kube-api-access-85b5v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.749001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.747917 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg"] Oct 30 18:04:28.845779 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.845704 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.846126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.845803 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85b5v\" (UniqueName: \"kubernetes.io/projected/9026e924-adb6-4a37-9e30-3ea721461bcd-kube-api-access-85b5v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.846126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.845912 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.846126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.846086 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzk7\" (UniqueName: \"kubernetes.io/projected/0590a1fb-5c50-48fd-8083-1db863e10ced-kube-api-access-kbzk7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.846394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.846137 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.846394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.846187 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.847053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.846994 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.848249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.848197 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.873332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.872905 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85b5v\" (UniqueName: \"kubernetes.io/projected/9026e924-adb6-4a37-9e30-3ea721461bcd-kube-api-access-85b5v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:28.902165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.901775 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" event={"ID":"6a1fa476-56bf-4fd4-a676-5da6109ab2ed","Type":"ContainerStarted","Data":"ade5fff8161b21658a2cce6cb131201d657c16cdbd1a46a1961cac196778a677"} Oct 30 18:04:28.912344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.912103 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" event={"ID":"28da4ab2-eb1e-458b-93a5-c063d30768de","Type":"ContainerStarted","Data":"4ae01d4b87285d7aafcb4504f3b064eb2cebfa5c15ca2201da02edcabb920309"} Oct 30 18:04:28.951050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.948454 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.951050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.948683 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzk7\" (UniqueName: \"kubernetes.io/projected/0590a1fb-5c50-48fd-8083-1db863e10ced-kube-api-access-kbzk7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.951050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.948747 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.951050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.949768 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.952566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.952433 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:28.976122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:28.975824 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzk7\" (UniqueName: \"kubernetes.io/projected/0590a1fb-5c50-48fd-8083-1db863e10ced-kube-api-access-kbzk7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:29.043097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:29.042448 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:04:29.071750 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:29.071654 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:04:29.455756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:29.454613 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5"] Oct 30 18:04:29.473395 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:29.473319 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9026e924_adb6_4a37_9e30_3ea721461bcd.slice/crio-dec055f2df21e485572a93c3aae49886ed7ab887c3665f0c31ea9015ff1ef4be WatchSource:0}: Error finding container dec055f2df21e485572a93c3aae49886ed7ab887c3665f0c31ea9015ff1ef4be: Status 404 returned error can't find the container with id dec055f2df21e485572a93c3aae49886ed7ab887c3665f0c31ea9015ff1ef4be Oct 30 18:04:29.490642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:29.490426 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg"] Oct 30 18:04:29.508795 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:04:29.508708 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0590a1fb_5c50_48fd_8083_1db863e10ced.slice/crio-963ac6707ed316dbcae77b5b1f4c7d46fdb09969ad80e297cd288429d6bdb6bf WatchSource:0}: Error finding container 963ac6707ed316dbcae77b5b1f4c7d46fdb09969ad80e297cd288429d6bdb6bf: Status 404 returned error can't find the container with id 963ac6707ed316dbcae77b5b1f4c7d46fdb09969ad80e297cd288429d6bdb6bf Oct 30 18:04:29.936288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:29.935830 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" event={"ID":"0590a1fb-5c50-48fd-8083-1db863e10ced","Type":"ContainerStarted","Data":"963ac6707ed316dbcae77b5b1f4c7d46fdb09969ad80e297cd288429d6bdb6bf"} Oct 30 18:04:29.953062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:29.952836 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" event={"ID":"9026e924-adb6-4a37-9e30-3ea721461bcd","Type":"ContainerStarted","Data":"dec055f2df21e485572a93c3aae49886ed7ab887c3665f0c31ea9015ff1ef4be"} Oct 30 18:04:30.829340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:30.829259 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:38.838142 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:38.837948 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:40.551535 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:40.551419 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:40.552224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:40.551607 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:40.552224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:40.551681 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:41.168291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:41.168213 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" event={"ID":"6a1fa476-56bf-4fd4-a676-5da6109ab2ed","Type":"ContainerStarted","Data":"3667e9f9db8e261d8a065b77e571d90433e0ae0d6bcd96b8bd8ce2c564242d61"} Oct 30 18:04:41.217031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:41.215885 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:41.217031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:41.216119 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:41.217031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:41.216181 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:04:41.240821 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:41.240661 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" podStartSLOduration=1.991366878 podStartE2EDuration="14.240618925s" totalImagesPullingTime="12.249252047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:41.224125475 +0000 UTC m=+175.612653594" watchObservedRunningTime="2026-10-30 18:04:41.240618925 +0000 UTC m=+175.629147039" Oct 30 18:04:41.867278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:41.867112 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:41.880377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:41.880309 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:42.192549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:42.192173 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" event={"ID":"28da4ab2-eb1e-458b-93a5-c063d30768de","Type":"ContainerStarted","Data":"747691b8f4a81b18fc3f0ae5750bc015f12b7f0c30115abef0eacdd26ab86854"} Oct 30 18:04:42.193076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:42.193027 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:42.196249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:42.196163 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn" Oct 30 18:04:42.241719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:42.240286 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podStartSLOduration=2.168883373 podStartE2EDuration="15.240238511s" totalImagesPullingTime="13.071355138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:42.239524279 +0000 UTC m=+176.628052392" watchObservedRunningTime="2026-10-30 18:04:42.240238511 +0000 UTC m=+176.628766634" Oct 30 18:04:42.709296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:42.704638 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:42.709296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:42.709095 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:42.719747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:42.719170 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:43.219559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:43.219502 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:04:46.272304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:46.272213 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" event={"ID":"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3","Type":"ContainerStarted","Data":"96a8e60bc5614fe630905b8ecc34a6e3f1c0d88fdc6cf4307dc28602d8edd5cb"} Oct 30 18:04:47.296353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:47.296262 5624 generic.go:356] "Generic (PLEG): container finished" podID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerID="96a8e60bc5614fe630905b8ecc34a6e3f1c0d88fdc6cf4307dc28602d8edd5cb" exitCode=0 Oct 30 18:04:47.297182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:47.296709 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" event={"ID":"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3","Type":"ContainerDied","Data":"96a8e60bc5614fe630905b8ecc34a6e3f1c0d88fdc6cf4307dc28602d8edd5cb"} Oct 30 18:04:47.306400 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:47.306321 5624 generic.go:356] "Generic (PLEG): container finished" podID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerID="aa0f9f44696cf91f48d5577b0e6173987ee16b6ab602fd41255ed28e7b1755c9" exitCode=0 Oct 30 18:04:47.306767 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:47.306429 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" event={"ID":"5d6bb2f7-7884-4bab-8bcb-65176afa9134","Type":"ContainerDied","Data":"aa0f9f44696cf91f48d5577b0e6173987ee16b6ab602fd41255ed28e7b1755c9"} Oct 30 18:04:48.339165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:48.337813 5624 generic.go:356] "Generic (PLEG): container finished" podID="404bc46d-2200-4328-9e0a-e82974d16395" containerID="bb4c7f17f4cb2182ceec3f3130afdfaeee8429bc06c469382298ac6a05093542" exitCode=0 Oct 30 18:04:48.339165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:48.338125 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" event={"ID":"404bc46d-2200-4328-9e0a-e82974d16395","Type":"ContainerDied","Data":"bb4c7f17f4cb2182ceec3f3130afdfaeee8429bc06c469382298ac6a05093542"} Oct 30 18:04:48.350186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:48.349368 5624 generic.go:356] "Generic (PLEG): container finished" podID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerID="bd68993033fe0e1ae94626f7301e0c4bd6e41bd79bf9eda5cab56fed58098b3d" exitCode=0 Oct 30 18:04:48.350186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:48.349635 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" event={"ID":"5bc8ce9e-74f2-4dd4-91d2-2a025720567c","Type":"ContainerDied","Data":"bd68993033fe0e1ae94626f7301e0c4bd6e41bd79bf9eda5cab56fed58098b3d"} Oct 30 18:04:48.361831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:48.361737 5624 generic.go:356] "Generic (PLEG): container finished" podID="843037eb-d856-4011-ab94-438be80ccb61" containerID="4443058a10269019c36875c6870df92e89394534ed3bfcc5d18f3cf7d994daf2" exitCode=0 Oct 30 18:04:48.362175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:48.361870 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" event={"ID":"843037eb-d856-4011-ab94-438be80ccb61","Type":"ContainerDied","Data":"4443058a10269019c36875c6870df92e89394534ed3bfcc5d18f3cf7d994daf2"} Oct 30 18:04:49.384566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:49.384146 5624 generic.go:356] "Generic (PLEG): container finished" podID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerID="2373cd81c7720c8f8f235aa1a053cad1ff7a1bb404cbc0bb2d209f5c38b6802d" exitCode=0 Oct 30 18:04:49.384566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:49.384448 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" event={"ID":"6606b549-4d64-49d2-9b98-cfce59198eb6","Type":"ContainerDied","Data":"2373cd81c7720c8f8f235aa1a053cad1ff7a1bb404cbc0bb2d209f5c38b6802d"} Oct 30 18:04:49.395062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:49.394983 5624 generic.go:356] "Generic (PLEG): container finished" podID="404bc46d-2200-4328-9e0a-e82974d16395" containerID="139b4c5e9209f95215da886184381ac3bfc97c98f0f47a69e12e54858332a4f0" exitCode=0 Oct 30 18:04:49.395425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:49.395322 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" event={"ID":"404bc46d-2200-4328-9e0a-e82974d16395","Type":"ContainerDied","Data":"139b4c5e9209f95215da886184381ac3bfc97c98f0f47a69e12e54858332a4f0"} Oct 30 18:04:49.410447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:49.410368 5624 generic.go:356] "Generic (PLEG): container finished" podID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerID="67e73a3e2ae8598a352b564cee8a58509e051419009a3d65ed6ca329f8e9c238" exitCode=0 Oct 30 18:04:49.410833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:49.410548 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" event={"ID":"5bc8ce9e-74f2-4dd4-91d2-2a025720567c","Type":"ContainerDied","Data":"67e73a3e2ae8598a352b564cee8a58509e051419009a3d65ed6ca329f8e9c238"} Oct 30 18:04:49.418817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:49.417268 5624 generic.go:356] "Generic (PLEG): container finished" podID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerID="f04d59aff95bd9191b2109d5b7f1f817bc286b96b04aa2f0faa690853d4ead0b" exitCode=0 Oct 30 18:04:49.418817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:49.417368 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" event={"ID":"5d6bb2f7-7884-4bab-8bcb-65176afa9134","Type":"ContainerDied","Data":"f04d59aff95bd9191b2109d5b7f1f817bc286b96b04aa2f0faa690853d4ead0b"} Oct 30 18:04:50.050535 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:04:50.050446 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:04:50.457598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:50.457522 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" event={"ID":"6606b549-4d64-49d2-9b98-cfce59198eb6","Type":"ContainerDied","Data":"9d026cc1d447d631319bbe3092c59420e6846e2879fc68cac755983e6ebbde46"} Oct 30 18:04:50.458417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:50.457536 5624 generic.go:356] "Generic (PLEG): container finished" podID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerID="9d026cc1d447d631319bbe3092c59420e6846e2879fc68cac755983e6ebbde46" exitCode=0 Oct 30 18:04:51.483174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:51.483039 5624 generic.go:356] "Generic (PLEG): container finished" podID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerID="c945841bb6ec4a5370bdbbb349ef42fa4dff437bfb6e4df8e9fc27ec3adebfee" exitCode=0 Oct 30 18:04:51.484094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:51.483341 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" event={"ID":"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1","Type":"ContainerDied","Data":"c945841bb6ec4a5370bdbbb349ef42fa4dff437bfb6e4df8e9fc27ec3adebfee"} Oct 30 18:04:52.500447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:52.500276 5624 generic.go:356] "Generic (PLEG): container finished" podID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerID="6cf0449b8b021d68cc33f54429533d309cdb80dcfe47945ede4b55a8eb60bf9a" exitCode=0 Oct 30 18:04:52.501205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:52.500502 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" event={"ID":"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1","Type":"ContainerDied","Data":"6cf0449b8b021d68cc33f54429533d309cdb80dcfe47945ede4b55a8eb60bf9a"} Oct 30 18:04:53.526910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:53.525416 5624 generic.go:356] "Generic (PLEG): container finished" podID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerID="f15efbef5a16378293e46a8537e105a5ade6181a068559d87c108f4b14d52b45" exitCode=0 Oct 30 18:04:53.526910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:53.525610 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" event={"ID":"0590a1fb-5c50-48fd-8083-1db863e10ced","Type":"ContainerDied","Data":"f15efbef5a16378293e46a8537e105a5ade6181a068559d87c108f4b14d52b45"} Oct 30 18:04:54.558624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:54.556037 5624 generic.go:356] "Generic (PLEG): container finished" podID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerID="08c9ac7184010dc3eae953279c34fffb102f460e701f8976d67d238a94797a37" exitCode=0 Oct 30 18:04:54.558624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:54.558073 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" event={"ID":"0590a1fb-5c50-48fd-8083-1db863e10ced","Type":"ContainerDied","Data":"08c9ac7184010dc3eae953279c34fffb102f460e701f8976d67d238a94797a37"} Oct 30 18:04:54.574503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:54.570676 5624 generic.go:356] "Generic (PLEG): container finished" podID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerID="0fd7148ad1217b7d85f9780c8e9bb9882b75be4af17e17ee01b015ebdb0c3376" exitCode=0 Oct 30 18:04:54.574503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:54.571143 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" event={"ID":"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3","Type":"ContainerDied","Data":"0fd7148ad1217b7d85f9780c8e9bb9882b75be4af17e17ee01b015ebdb0c3376"} Oct 30 18:04:54.604764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:54.604030 5624 generic.go:356] "Generic (PLEG): container finished" podID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerID="62aea363403ed7f9a19cb43e4e600d4375ac49d3b0d47666b0a0e2129a1a36fc" exitCode=0 Oct 30 18:04:54.604764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:54.604179 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" event={"ID":"9026e924-adb6-4a37-9e30-3ea721461bcd","Type":"ContainerDied","Data":"62aea363403ed7f9a19cb43e4e600d4375ac49d3b0d47666b0a0e2129a1a36fc"} Oct 30 18:04:55.629478 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:55.629390 5624 generic.go:356] "Generic (PLEG): container finished" podID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerID="361c41c47a625591b1af20e98e7fecd085aa79598047b27022141317e3ea4eb0" exitCode=0 Oct 30 18:04:55.630397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:55.629700 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" event={"ID":"9026e924-adb6-4a37-9e30-3ea721461bcd","Type":"ContainerDied","Data":"361c41c47a625591b1af20e98e7fecd085aa79598047b27022141317e3ea4eb0"} Oct 30 18:04:56.686296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:56.686202 5624 generic.go:356] "Generic (PLEG): container finished" podID="843037eb-d856-4011-ab94-438be80ccb61" containerID="def23bc1c2a05b2e568e3053989ac707979cd00fef199e208def2a03d0ed182c" exitCode=0 Oct 30 18:04:56.687237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:04:56.686417 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" event={"ID":"843037eb-d856-4011-ab94-438be80ccb61","Type":"ContainerDied","Data":"def23bc1c2a05b2e568e3053989ac707979cd00fef199e208def2a03d0ed182c"} Oct 30 18:05:12.989256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:12.989173 5624 generic.go:356] "Generic (PLEG): container finished" podID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerID="8453535f35b7d46a4826d70b9e603f2164da34b40500288c595384785955fc72" exitCode=0 Oct 30 18:05:12.990162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:12.989432 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" event={"ID":"5bc8ce9e-74f2-4dd4-91d2-2a025720567c","Type":"ContainerDied","Data":"8453535f35b7d46a4826d70b9e603f2164da34b40500288c595384785955fc72"} Oct 30 18:05:13.008176 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:13.008080 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" event={"ID":"5d6bb2f7-7884-4bab-8bcb-65176afa9134","Type":"ContainerStarted","Data":"8c9c992fd68cffa2f757e8e572298e763e3002395bc915c656d51e42df5dc8d4"} Oct 30 18:05:13.019494 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:13.019416 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" event={"ID":"6606b549-4d64-49d2-9b98-cfce59198eb6","Type":"ContainerStarted","Data":"53c65edc276fb8dce0a818f909566665da0d678b7ae5e492e22cf61bae21e333"} Oct 30 18:05:13.036662 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:13.036586 5624 generic.go:356] "Generic (PLEG): container finished" podID="404bc46d-2200-4328-9e0a-e82974d16395" containerID="3362e509dc9b105ffc72f7ce47094ca8b38345f7e391d441c219dfadd919f513" exitCode=0 Oct 30 18:05:13.037070 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:13.036689 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" event={"ID":"404bc46d-2200-4328-9e0a-e82974d16395","Type":"ContainerDied","Data":"3362e509dc9b105ffc72f7ce47094ca8b38345f7e391d441c219dfadd919f513"} Oct 30 18:05:13.080974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:13.080115 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" podStartSLOduration=3.458141933 podStartE2EDuration="51.080071817s" totalImagesPullingTime="47.621929884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:13.078755966 +0000 UTC m=+207.467284096" watchObservedRunningTime="2026-10-30 18:05:13.080071817 +0000 UTC m=+207.468599943" Oct 30 18:05:13.123608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:13.123466 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" podStartSLOduration=3.70894818 podStartE2EDuration="51.123426858s" totalImagesPullingTime="47.414478678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:13.121227002 +0000 UTC m=+207.509755120" watchObservedRunningTime="2026-10-30 18:05:13.123426858 +0000 UTC m=+207.511954973" Oct 30 18:05:14.062304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.062213 5624 generic.go:356] "Generic (PLEG): container finished" podID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerID="8c9c992fd68cffa2f757e8e572298e763e3002395bc915c656d51e42df5dc8d4" exitCode=0 Oct 30 18:05:14.063617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.062655 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" event={"ID":"5d6bb2f7-7884-4bab-8bcb-65176afa9134","Type":"ContainerDied","Data":"8c9c992fd68cffa2f757e8e572298e763e3002395bc915c656d51e42df5dc8d4"} Oct 30 18:05:14.080577 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.079879 5624 generic.go:356] "Generic (PLEG): container finished" podID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerID="53c65edc276fb8dce0a818f909566665da0d678b7ae5e492e22cf61bae21e333" exitCode=0 Oct 30 18:05:14.080577 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.080086 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" event={"ID":"6606b549-4d64-49d2-9b98-cfce59198eb6","Type":"ContainerDied","Data":"53c65edc276fb8dce0a818f909566665da0d678b7ae5e492e22cf61bae21e333"} Oct 30 18:05:14.406274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.406204 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:05:14.462340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.462270 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:05:14.515164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.514658 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-kube-api-access-ztxbl\" (UniqueName: \"kubernetes.io/projected/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-kube-api-access-ztxbl\") pod \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " Oct 30 18:05:14.515164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.514786 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-bundle\") pod \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " Oct 30 18:05:14.515164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.514863 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-util\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-util\") pod \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\" (UID: \"5bc8ce9e-74f2-4dd4-91d2-2a025720567c\") " Oct 30 18:05:14.517611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.517486 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-bundle" pod "5bc8ce9e-74f2-4dd4-91d2-2a025720567c" (UID: "5bc8ce9e-74f2-4dd4-91d2-2a025720567c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.518663 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.518595 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.527043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.526907 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-util" pod "5bc8ce9e-74f2-4dd4-91d2-2a025720567c" (UID: "5bc8ce9e-74f2-4dd4-91d2-2a025720567c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.529565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.529403 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-kube-api-access-ztxbl" pod "5bc8ce9e-74f2-4dd4-91d2-2a025720567c" (UID: "5bc8ce9e-74f2-4dd4-91d2-2a025720567c"). InnerVolumeSpecName "kube-api-access-ztxbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:14.621635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.620109 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-util\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-util\") pod \"404bc46d-2200-4328-9e0a-e82974d16395\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " Oct 30 18:05:14.621635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.620447 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404bc46d-2200-4328-9e0a-e82974d16395-kube-api-access-s5pwg\" (UniqueName: \"kubernetes.io/projected/404bc46d-2200-4328-9e0a-e82974d16395-kube-api-access-s5pwg\") pod \"404bc46d-2200-4328-9e0a-e82974d16395\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " Oct 30 18:05:14.621635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.620523 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-bundle\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-bundle\") pod \"404bc46d-2200-4328-9e0a-e82974d16395\" (UID: \"404bc46d-2200-4328-9e0a-e82974d16395\") " Oct 30 18:05:14.621635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.621016 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztxbl\" (UniqueName: \"kubernetes.io/projected/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-kube-api-access-ztxbl\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.621635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.621043 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bc8ce9e-74f2-4dd4-91d2-2a025720567c-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.626299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.626176 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-bundle" pod "404bc46d-2200-4328-9e0a-e82974d16395" (UID: "404bc46d-2200-4328-9e0a-e82974d16395"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.629154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.629028 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404bc46d-2200-4328-9e0a-e82974d16395-kube-api-access-s5pwg" pod "404bc46d-2200-4328-9e0a-e82974d16395" (UID: "404bc46d-2200-4328-9e0a-e82974d16395"). InnerVolumeSpecName "kube-api-access-s5pwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:14.631947 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.631833 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-util" pod "404bc46d-2200-4328-9e0a-e82974d16395" (UID: "404bc46d-2200-4328-9e0a-e82974d16395"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:14.722345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.722269 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.722345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.722343 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5pwg\" (UniqueName: \"kubernetes.io/projected/404bc46d-2200-4328-9e0a-e82974d16395-kube-api-access-s5pwg\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:14.722345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:14.722365 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/404bc46d-2200-4328-9e0a-e82974d16395-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:15.100615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.100539 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" event={"ID":"404bc46d-2200-4328-9e0a-e82974d16395","Type":"ContainerDied","Data":"72905cb7b825b671860cef06f9679b917cbd1cacd006b5bb9e5aa1e110699d39"} Oct 30 18:05:15.101341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.100646 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72905cb7b825b671860cef06f9679b917cbd1cacd006b5bb9e5aa1e110699d39" Oct 30 18:05:15.101341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.100766 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2sdrv" Oct 30 18:05:15.122485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.122418 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" event={"ID":"5bc8ce9e-74f2-4dd4-91d2-2a025720567c","Type":"ContainerDied","Data":"7b1267ae89f40c10666b4154cb9c840ca1d22e44670d4cc54e7306bf12269ca5"} Oct 30 18:05:15.122485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.122459 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6plp4z" Oct 30 18:05:15.122917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.122518 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b1267ae89f40c10666b4154cb9c840ca1d22e44670d4cc54e7306bf12269ca5" Oct 30 18:05:15.134034 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.132129 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" event={"ID":"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1","Type":"ContainerStarted","Data":"5c236aad9f9d643c755d6fcb3ec0bf741b9d3a7de617513de6fe1b3db54d4e52"} Oct 30 18:05:15.181985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.181822 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" podStartSLOduration=2.930499517 podStartE2EDuration="49.181786508s" totalImagesPullingTime="46.251286991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:15.178132684 +0000 UTC m=+209.566660801" watchObservedRunningTime="2026-10-30 18:05:15.181786508 +0000 UTC m=+209.570314629" Oct 30 18:05:15.406707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.406513 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:05:15.491267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.491187 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:05:15.537309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.536146 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-bundle\") pod \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " Oct 30 18:05:15.537309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.536254 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6bb2f7-7884-4bab-8bcb-65176afa9134-kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/5d6bb2f7-7884-4bab-8bcb-65176afa9134-kube-api-access-9nps7\") pod \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " Oct 30 18:05:15.537309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.536455 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-util\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-util\") pod \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\" (UID: \"5d6bb2f7-7884-4bab-8bcb-65176afa9134\") " Oct 30 18:05:15.539049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.537854 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-bundle" pod "5d6bb2f7-7884-4bab-8bcb-65176afa9134" (UID: "5d6bb2f7-7884-4bab-8bcb-65176afa9134"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:15.544219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.544055 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6bb2f7-7884-4bab-8bcb-65176afa9134-kube-api-access-9nps7" pod "5d6bb2f7-7884-4bab-8bcb-65176afa9134" (UID: "5d6bb2f7-7884-4bab-8bcb-65176afa9134"). InnerVolumeSpecName "kube-api-access-9nps7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:15.547469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.547380 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-util" pod "5d6bb2f7-7884-4bab-8bcb-65176afa9134" (UID: "5d6bb2f7-7884-4bab-8bcb-65176afa9134"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:15.637797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.637730 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6606b549-4d64-49d2-9b98-cfce59198eb6-kube-api-access-sxghs\" (UniqueName: \"kubernetes.io/projected/6606b549-4d64-49d2-9b98-cfce59198eb6-kube-api-access-sxghs\") pod \"6606b549-4d64-49d2-9b98-cfce59198eb6\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " Oct 30 18:05:15.638168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.637853 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-util\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-util\") pod \"6606b549-4d64-49d2-9b98-cfce59198eb6\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " Oct 30 18:05:15.638168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.637921 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-bundle\") pod \"6606b549-4d64-49d2-9b98-cfce59198eb6\" (UID: \"6606b549-4d64-49d2-9b98-cfce59198eb6\") " Oct 30 18:05:15.639573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.638387 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:15.639573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.638418 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d6bb2f7-7884-4bab-8bcb-65176afa9134-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:15.639573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.638439 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nps7\" (UniqueName: \"kubernetes.io/projected/5d6bb2f7-7884-4bab-8bcb-65176afa9134-kube-api-access-9nps7\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:15.640960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.640827 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-bundle" pod "6606b549-4d64-49d2-9b98-cfce59198eb6" (UID: "6606b549-4d64-49d2-9b98-cfce59198eb6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:15.645997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.645878 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-util" pod "6606b549-4d64-49d2-9b98-cfce59198eb6" (UID: "6606b549-4d64-49d2-9b98-cfce59198eb6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:15.656482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.656388 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6606b549-4d64-49d2-9b98-cfce59198eb6-kube-api-access-sxghs" pod "6606b549-4d64-49d2-9b98-cfce59198eb6" (UID: "6606b549-4d64-49d2-9b98-cfce59198eb6"). InnerVolumeSpecName "kube-api-access-sxghs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:15.739788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.739638 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:15.739788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.739693 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxghs\" (UniqueName: \"kubernetes.io/projected/6606b549-4d64-49d2-9b98-cfce59198eb6-kube-api-access-sxghs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:15.739788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:15.739714 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6606b549-4d64-49d2-9b98-cfce59198eb6-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:16.159965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:16.158850 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" event={"ID":"6606b549-4d64-49d2-9b98-cfce59198eb6","Type":"ContainerDied","Data":"70dfb7356a6eba04fbe51530181e7dd4604bd69fd0120e0b3f39c9cf4990a2e6"} Oct 30 18:05:16.159965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:16.158999 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70dfb7356a6eba04fbe51530181e7dd4604bd69fd0120e0b3f39c9cf4990a2e6" Oct 30 18:05:16.159965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:16.159156 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nktvt" Oct 30 18:05:16.165788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:16.165734 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" Oct 30 18:05:16.166124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:16.165745 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax6gpl" event={"ID":"5d6bb2f7-7884-4bab-8bcb-65176afa9134","Type":"ContainerDied","Data":"365dd00898fe1a45c932a766c6a931f23369100be76473c333eca6833f7dd93d"} Oct 30 18:05:16.166124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:16.165909 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="365dd00898fe1a45c932a766c6a931f23369100be76473c333eca6833f7dd93d" Oct 30 18:05:16.189511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:16.189416 5624 generic.go:356] "Generic (PLEG): container finished" podID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerID="5c236aad9f9d643c755d6fcb3ec0bf741b9d3a7de617513de6fe1b3db54d4e52" exitCode=0 Oct 30 18:05:16.189824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:16.189541 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" event={"ID":"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1","Type":"ContainerDied","Data":"5c236aad9f9d643c755d6fcb3ec0bf741b9d3a7de617513de6fe1b3db54d4e52"} Oct 30 18:05:17.203516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.203430 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" event={"ID":"0590a1fb-5c50-48fd-8083-1db863e10ced","Type":"ContainerStarted","Data":"3bc553d3e3a571da0a97a723e0b1784ae0fcd06d1c7e755f220050408002619e"} Oct 30 18:05:17.208798 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.208735 5624 generic.go:356] "Generic (PLEG): container finished" podID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerID="2e439bd6a659c470a8a1e6ee2aafa1bff7f8aa0b30b6d85ae32c64bd0413f0f9" exitCode=0 Oct 30 18:05:17.209189 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.208805 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" event={"ID":"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3","Type":"ContainerDied","Data":"2e439bd6a659c470a8a1e6ee2aafa1bff7f8aa0b30b6d85ae32c64bd0413f0f9"} Oct 30 18:05:17.250828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.250729 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" podStartSLOduration=3.235215306 podStartE2EDuration="49.250696627s" totalImagesPullingTime="46.015481321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:17.2459777 +0000 UTC m=+211.634505819" watchObservedRunningTime="2026-10-30 18:05:17.250696627 +0000 UTC m=+211.639224744" Oct 30 18:05:17.441158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.441091 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:05:17.562831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.562803 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-kube-api-access-5qsml\" (UniqueName: \"kubernetes.io/projected/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-kube-api-access-5qsml\") pod \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " Oct 30 18:05:17.562986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.562925 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-bundle\") pod \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " Oct 30 18:05:17.563074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.563020 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-util\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-util\") pod \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\" (UID: \"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1\") " Oct 30 18:05:17.563908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.563850 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-bundle" pod "98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" (UID: "98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:17.566745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.566657 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-kube-api-access-5qsml" pod "98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" (UID: "98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1"). InnerVolumeSpecName "kube-api-access-5qsml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:17.571581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.571496 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-util" pod "98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" (UID: "98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:17.664984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.664893 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qsml\" (UniqueName: \"kubernetes.io/projected/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-kube-api-access-5qsml\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:17.664984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.664973 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:17.664984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:17.664985 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:18.222424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.222234 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" event={"ID":"98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1","Type":"ContainerDied","Data":"ced1e08f39449cf097cdd6f71d7b182679efa9561ab9376a8b8afc2bb85e2a7b"} Oct 30 18:05:18.222424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.222351 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced1e08f39449cf097cdd6f71d7b182679efa9561ab9376a8b8afc2bb85e2a7b" Oct 30 18:05:18.222424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.222273 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71wtd4h" Oct 30 18:05:18.227655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.227583 5624 generic.go:356] "Generic (PLEG): container finished" podID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerID="25c94eaceb6c5ce0f9a1825f746051f8458abb557b49c9eec6b1f11fba683f62" exitCode=0 Oct 30 18:05:18.227994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.227740 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" event={"ID":"9026e924-adb6-4a37-9e30-3ea721461bcd","Type":"ContainerDied","Data":"25c94eaceb6c5ce0f9a1825f746051f8458abb557b49c9eec6b1f11fba683f62"} Oct 30 18:05:18.232408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.232336 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" event={"ID":"843037eb-d856-4011-ab94-438be80ccb61","Type":"ContainerStarted","Data":"95e4081c540129c1e39618b0a63357268f361715b0059453444f603c460fc260"} Oct 30 18:05:18.238083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.237900 5624 generic.go:356] "Generic (PLEG): container finished" podID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerID="3bc553d3e3a571da0a97a723e0b1784ae0fcd06d1c7e755f220050408002619e" exitCode=0 Oct 30 18:05:18.238083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.238047 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" event={"ID":"0590a1fb-5c50-48fd-8083-1db863e10ced","Type":"ContainerDied","Data":"3bc553d3e3a571da0a97a723e0b1784ae0fcd06d1c7e755f220050408002619e"} Oct 30 18:05:18.386639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.386547 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" podStartSLOduration=3.243848973 podStartE2EDuration="55.386514606s" totalImagesPullingTime="52.142665633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:18.385406945 +0000 UTC m=+212.773935062" watchObservedRunningTime="2026-10-30 18:05:18.386514606 +0000 UTC m=+212.775042722" Oct 30 18:05:18.460986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.460864 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:05:18.474566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.474437 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-kube-api-access-h2cch\" (UniqueName: \"kubernetes.io/projected/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-kube-api-access-h2cch\") pod \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " Oct 30 18:05:18.474566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.474552 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-util\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-util\") pod \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " Oct 30 18:05:18.474852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.474594 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-bundle\") pod \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\" (UID: \"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3\") " Oct 30 18:05:18.475264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.475223 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-bundle" pod "9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" (UID: "9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:18.478260 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.478189 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-kube-api-access-h2cch" pod "9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" (UID: "9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3"). InnerVolumeSpecName "kube-api-access-h2cch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:18.483347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.483270 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-util" pod "9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" (UID: "9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:18.575508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.575426 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:18.575508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.575498 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2cch\" (UniqueName: \"kubernetes.io/projected/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-kube-api-access-h2cch\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:18.575508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:18.575517 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:19.250307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.250238 5624 generic.go:356] "Generic (PLEG): container finished" podID="843037eb-d856-4011-ab94-438be80ccb61" containerID="95e4081c540129c1e39618b0a63357268f361715b0059453444f603c460fc260" exitCode=0 Oct 30 18:05:19.251148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.250327 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" event={"ID":"843037eb-d856-4011-ab94-438be80ccb61","Type":"ContainerDied","Data":"95e4081c540129c1e39618b0a63357268f361715b0059453444f603c460fc260"} Oct 30 18:05:19.254180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.254116 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" event={"ID":"9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3","Type":"ContainerDied","Data":"492a45a2398fbbeb3319fa07bd3741700f7ef9d42166022e88dbc784af240223"} Oct 30 18:05:19.254534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.254225 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="492a45a2398fbbeb3319fa07bd3741700f7ef9d42166022e88dbc784af240223" Oct 30 18:05:19.254534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.254225 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448ccg8" Oct 30 18:05:19.440657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.440593 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:19.489626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.489560 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:05:19.496245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.496181 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:05:19.588653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.588585 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-bundle\") pod \"0590a1fb-5c50-48fd-8083-1db863e10ced\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " Oct 30 18:05:19.588653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.588656 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-bundle\") pod \"9026e924-adb6-4a37-9e30-3ea721461bcd\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " Oct 30 18:05:19.589135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.588734 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-util\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-util\") pod \"0590a1fb-5c50-48fd-8083-1db863e10ced\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " Oct 30 18:05:19.589135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.588801 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9026e924-adb6-4a37-9e30-3ea721461bcd-kube-api-access-85b5v\" (UniqueName: \"kubernetes.io/projected/9026e924-adb6-4a37-9e30-3ea721461bcd-kube-api-access-85b5v\") pod \"9026e924-adb6-4a37-9e30-3ea721461bcd\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " Oct 30 18:05:19.589135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.588875 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0590a1fb-5c50-48fd-8083-1db863e10ced-kube-api-access-kbzk7\" (UniqueName: \"kubernetes.io/projected/0590a1fb-5c50-48fd-8083-1db863e10ced-kube-api-access-kbzk7\") pod \"0590a1fb-5c50-48fd-8083-1db863e10ced\" (UID: \"0590a1fb-5c50-48fd-8083-1db863e10ced\") " Oct 30 18:05:19.589135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.588964 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-util\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-util\") pod \"9026e924-adb6-4a37-9e30-3ea721461bcd\" (UID: \"9026e924-adb6-4a37-9e30-3ea721461bcd\") " Oct 30 18:05:19.589661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.589597 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-bundle" pod "9026e924-adb6-4a37-9e30-3ea721461bcd" (UID: "9026e924-adb6-4a37-9e30-3ea721461bcd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:19.589919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.589875 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-bundle" pod "0590a1fb-5c50-48fd-8083-1db863e10ced" (UID: "0590a1fb-5c50-48fd-8083-1db863e10ced"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:19.593124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.593014 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0590a1fb-5c50-48fd-8083-1db863e10ced-kube-api-access-kbzk7" pod "0590a1fb-5c50-48fd-8083-1db863e10ced" (UID: "0590a1fb-5c50-48fd-8083-1db863e10ced"). InnerVolumeSpecName "kube-api-access-kbzk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:19.593124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.593095 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9026e924-adb6-4a37-9e30-3ea721461bcd-kube-api-access-85b5v" pod "9026e924-adb6-4a37-9e30-3ea721461bcd" (UID: "9026e924-adb6-4a37-9e30-3ea721461bcd"). InnerVolumeSpecName "kube-api-access-85b5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:19.599450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.599352 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-util" pod "9026e924-adb6-4a37-9e30-3ea721461bcd" (UID: "9026e924-adb6-4a37-9e30-3ea721461bcd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:19.601541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.601491 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-util" pod "0590a1fb-5c50-48fd-8083-1db863e10ced" (UID: "0590a1fb-5c50-48fd-8083-1db863e10ced"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:19.607313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.607241 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:19.690362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.690248 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:19.690362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.690326 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85b5v\" (UniqueName: \"kubernetes.io/projected/9026e924-adb6-4a37-9e30-3ea721461bcd-kube-api-access-85b5v\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:19.690362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.690358 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbzk7\" (UniqueName: \"kubernetes.io/projected/0590a1fb-5c50-48fd-8083-1db863e10ced-kube-api-access-kbzk7\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:19.690362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.690388 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:19.690847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.690408 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0590a1fb-5c50-48fd-8083-1db863e10ced-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:19.690847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:19.690425 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9026e924-adb6-4a37-9e30-3ea721461bcd-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:19.930154 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:05:19.930087 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:05:20.266237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.266182 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" Oct 30 18:05:20.267064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.266181 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c64rxc5" event={"ID":"9026e924-adb6-4a37-9e30-3ea721461bcd","Type":"ContainerDied","Data":"dec055f2df21e485572a93c3aae49886ed7ab887c3665f0c31ea9015ff1ef4be"} Oct 30 18:05:20.267064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.266344 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dec055f2df21e485572a93c3aae49886ed7ab887c3665f0c31ea9015ff1ef4be" Oct 30 18:05:20.270532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.270474 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" event={"ID":"0590a1fb-5c50-48fd-8083-1db863e10ced","Type":"ContainerDied","Data":"963ac6707ed316dbcae77b5b1f4c7d46fdb09969ad80e297cd288429d6bdb6bf"} Oct 30 18:05:20.270848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.270569 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="963ac6707ed316dbcae77b5b1f4c7d46fdb09969ad80e297cd288429d6bdb6bf" Oct 30 18:05:20.270848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.270493 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847advtqg" Oct 30 18:05:20.385550 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.385478 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:20.457804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.457741 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:05:20.499700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.499621 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-bundle\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-bundle\") pod \"843037eb-d856-4011-ab94-438be80ccb61\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " Oct 30 18:05:20.500081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.499774 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-util\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-util\") pod \"843037eb-d856-4011-ab94-438be80ccb61\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " Oct 30 18:05:20.500081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.499808 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843037eb-d856-4011-ab94-438be80ccb61-kube-api-access-qtws7\" (UniqueName: \"kubernetes.io/projected/843037eb-d856-4011-ab94-438be80ccb61-kube-api-access-qtws7\") pod \"843037eb-d856-4011-ab94-438be80ccb61\" (UID: \"843037eb-d856-4011-ab94-438be80ccb61\") " Oct 30 18:05:20.500673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.500621 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-bundle" pod "843037eb-d856-4011-ab94-438be80ccb61" (UID: "843037eb-d856-4011-ab94-438be80ccb61"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:20.504875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.504800 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843037eb-d856-4011-ab94-438be80ccb61-kube-api-access-qtws7" pod "843037eb-d856-4011-ab94-438be80ccb61" (UID: "843037eb-d856-4011-ab94-438be80ccb61"). InnerVolumeSpecName "kube-api-access-qtws7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:05:20.510207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.510128 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-util" pod "843037eb-d856-4011-ab94-438be80ccb61" (UID: "843037eb-d856-4011-ab94-438be80ccb61"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:05:20.537718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.537506 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz"] Oct 30 18:05:20.538289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538205 5624 cpu_manager.go:414] "containerMap: removing container" podUID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerName="pull" Oct 30 18:05:20.538289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538237 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerName="pull" Oct 30 18:05:20.538289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538253 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerName="util" Oct 30 18:05:20.538289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538260 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerName="util" Oct 30 18:05:20.538289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538276 5624 cpu_manager.go:414] "containerMap: removing container" podUID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerName="extract" Oct 30 18:05:20.538289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538286 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538316 5624 cpu_manager.go:414] "containerMap: removing container" podUID="404bc46d-2200-4328-9e0a-e82974d16395" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538324 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404bc46d-2200-4328-9e0a-e82974d16395" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538336 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538344 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538363 5624 cpu_manager.go:414] "containerMap: removing container" podUID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538368 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538376 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538383 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538395 5624 cpu_manager.go:414] "containerMap: removing container" podUID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538403 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538421 5624 cpu_manager.go:414] "containerMap: removing container" podUID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538434 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538444 5624 cpu_manager.go:414] "containerMap: removing container" podUID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538455 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538468 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538475 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538490 5624 cpu_manager.go:414] "containerMap: removing container" podUID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538498 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538514 5624 cpu_manager.go:414] "containerMap: removing container" podUID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538522 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538535 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538544 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538552 5624 cpu_manager.go:414] "containerMap: removing container" podUID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538557 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538569 5624 cpu_manager.go:414] "containerMap: removing container" podUID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538573 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538580 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538586 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538593 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538599 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538616 5624 cpu_manager.go:414] "containerMap: removing container" podUID="404bc46d-2200-4328-9e0a-e82974d16395" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538622 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404bc46d-2200-4328-9e0a-e82974d16395" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538634 5624 cpu_manager.go:414] "containerMap: removing container" podUID="843037eb-d856-4011-ab94-438be80ccb61" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538639 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843037eb-d856-4011-ab94-438be80ccb61" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538647 5624 cpu_manager.go:414] "containerMap: removing container" podUID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538655 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerName="util" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538662 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538667 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538680 5624 cpu_manager.go:414] "containerMap: removing container" podUID="843037eb-d856-4011-ab94-438be80ccb61" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538687 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843037eb-d856-4011-ab94-438be80ccb61" containerName="extract" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538699 5624 cpu_manager.go:414] "containerMap: removing container" podUID="843037eb-d856-4011-ab94-438be80ccb61" containerName="pull" Oct 30 18:05:20.538708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538734 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843037eb-d856-4011-ab94-438be80ccb61" containerName="pull" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538755 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerName="util" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538761 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerName="util" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538768 5624 cpu_manager.go:414] "containerMap: removing container" podUID="404bc46d-2200-4328-9e0a-e82974d16395" containerName="pull" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538774 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404bc46d-2200-4328-9e0a-e82974d16395" containerName="pull" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538780 5624 cpu_manager.go:414] "containerMap: removing container" podUID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538787 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538907 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc8ce9e-74f2-4dd4-91d2-2a025720567c" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538924 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c4271e-b155-4c31-ba6d-e7d7c4d7a9c1" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538953 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="6606b549-4d64-49d2-9b98-cfce59198eb6" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538963 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="9026e924-adb6-4a37-9e30-3ea721461bcd" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538973 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="404bc46d-2200-4328-9e0a-e82974d16395" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538982 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ddc67f4-2b56-40b9-ac6b-7fae3444d1c3" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.538992 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="843037eb-d856-4011-ab94-438be80ccb61" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.539003 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6bb2f7-7884-4bab-8bcb-65176afa9134" containerName="extract" Oct 30 18:05:20.540130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.539013 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="0590a1fb-5c50-48fd-8083-1db863e10ced" containerName="extract" Oct 30 18:05:20.543426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.543374 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.578559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.578493 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:05:20.578886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.578754 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:05:20.578886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.578842 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:05:20.578886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.578878 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:05:20.579076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.579057 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwrgm\"" Oct 30 18:05:20.579125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.579084 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:05:20.601320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601263 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.601320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601336 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92cd679c-b797-4195-aae4-5312581fbce9-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.601773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601381 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htfnm\" (UniqueName: \"kubernetes.io/projected/92cd679c-b797-4195-aae4-5312581fbce9-kube-api-access-htfnm\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.601773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601437 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.601773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601510 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.601773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601542 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92cd679c-b797-4195-aae4-5312581fbce9-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.601773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601629 5624 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-bundle\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:20.601773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601644 5624 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/843037eb-d856-4011-ab94-438be80ccb61-util\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:20.601773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.601661 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtws7\" (UniqueName: \"kubernetes.io/projected/843037eb-d856-4011-ab94-438be80ccb61-kube-api-access-qtws7\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:05:20.604380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.604308 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz"] Oct 30 18:05:20.702901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.702733 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.702901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.702849 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.702901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.702895 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92cd679c-b797-4195-aae4-5312581fbce9-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.703464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.703023 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.703464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.703074 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92cd679c-b797-4195-aae4-5312581fbce9-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.703464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.703113 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htfnm\" (UniqueName: \"kubernetes.io/projected/92cd679c-b797-4195-aae4-5312581fbce9-kube-api-access-htfnm\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.703844 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.703799 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/92cd679c-b797-4195-aae4-5312581fbce9-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.707268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.707199 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92cd679c-b797-4195-aae4-5312581fbce9-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.707569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.707329 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.707569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.707329 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.707569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.707473 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/92cd679c-b797-4195-aae4-5312581fbce9-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.724200 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.724128 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htfnm\" (UniqueName: \"kubernetes.io/projected/92cd679c-b797-4195-aae4-5312581fbce9-kube-api-access-htfnm\") pod \"opendatahub-operator-controller-manager-85c77bd65b-s7xqz\" (UID: \"92cd679c-b797-4195-aae4-5312581fbce9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:20.746163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.746089 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:20.867097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.866815 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:20.867658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:20.867619 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:21.100668 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:21.100602 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:21.256482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:21.256429 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:21.280530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:21.279804 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz"] Oct 30 18:05:21.283290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:21.283247 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" Oct 30 18:05:21.283547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:21.283245 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651csvqhq" event={"ID":"843037eb-d856-4011-ab94-438be80ccb61","Type":"ContainerDied","Data":"37446d1122f95d30503bad8fdb11ea962c6123d9bb18ae0fff99d547a7a264b0"} Oct 30 18:05:21.283547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:21.283409 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37446d1122f95d30503bad8fdb11ea962c6123d9bb18ae0fff99d547a7a264b0" Oct 30 18:05:21.297061 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:05:21.297004 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cd679c_b797_4195_aae4_5312581fbce9.slice/crio-786f9f36fd5cf4a949328549cc86920de37e0ebc1640e9d422b009ef42c02b6e WatchSource:0}: Error finding container 786f9f36fd5cf4a949328549cc86920de37e0ebc1640e9d422b009ef42c02b6e: Status 404 returned error can't find the container with id 786f9f36fd5cf4a949328549cc86920de37e0ebc1640e9d422b009ef42c02b6e Oct 30 18:05:22.298134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:22.298060 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" event={"ID":"92cd679c-b797-4195-aae4-5312581fbce9","Type":"ContainerStarted","Data":"786f9f36fd5cf4a949328549cc86920de37e0ebc1640e9d422b009ef42c02b6e"} Oct 30 18:05:23.457593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:23.457514 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:23.613728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:23.613655 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:25.323722 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:25.323646 5624 generic.go:356] "Generic (PLEG): container finished" podID="92cd679c-b797-4195-aae4-5312581fbce9" containerID="9ba46d854945ea5cd1f85d82cf7f6dd29186e3086c7db2c39a103105c87fae21" exitCode=0 Oct 30 18:05:25.324403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:25.323742 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" event={"ID":"92cd679c-b797-4195-aae4-5312581fbce9","Type":"ContainerDied","Data":"9ba46d854945ea5cd1f85d82cf7f6dd29186e3086c7db2c39a103105c87fae21"} Oct 30 18:05:26.335253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:26.335170 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" event={"ID":"92cd679c-b797-4195-aae4-5312581fbce9","Type":"ContainerStarted","Data":"50cbeed015c99eb84f2996fa626ee48e3bb8466c8d9523c7d76b322be3d15461"} Oct 30 18:05:26.335997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:26.335350 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:26.425957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:26.425837 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" podStartSLOduration=2.129761425 podStartE2EDuration="6.425802525s" totalImagesPullingTime="3.553181931s" totalInitContainerRuntime="742.859169ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:26.412125458 +0000 UTC m=+220.800653575" watchObservedRunningTime="2026-10-30 18:05:26.425802525 +0000 UTC m=+220.814330645" Oct 30 18:05:26.701103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:26.701039 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:26.827517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:26.827456 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:27.061916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:27.061740 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:29.528057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:29.527977 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:29.638454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:29.638351 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:30.064440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.064372 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:30.202769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.202692 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:30.512999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.512904 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt"] Oct 30 18:05:30.521813 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.521779 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:05:30.539381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.539307 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt"] Oct 30 18:05:30.540750 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.540700 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:05:30.542629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.542564 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 18:05:30.548728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.548674 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:05:30.549080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.548806 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2x585\"" Oct 30 18:05:30.625635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.625547 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67tlx\" (UniqueName: \"kubernetes.io/projected/75ee8c7b-b877-4f5c-9b74-08bda044c20c-kube-api-access-67tlx\") pod \"dns-operator-controller-manager-8557cbdb84-kl2vt\" (UID: \"75ee8c7b-b877-4f5c-9b74-08bda044c20c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:05:30.727288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.727213 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67tlx\" (UniqueName: \"kubernetes.io/projected/75ee8c7b-b877-4f5c-9b74-08bda044c20c-kube-api-access-67tlx\") pod \"dns-operator-controller-manager-8557cbdb84-kl2vt\" (UID: \"75ee8c7b-b877-4f5c-9b74-08bda044c20c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:05:30.764097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.763902 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67tlx\" (UniqueName: \"kubernetes.io/projected/75ee8c7b-b877-4f5c-9b74-08bda044c20c-kube-api-access-67tlx\") pod \"dns-operator-controller-manager-8557cbdb84-kl2vt\" (UID: \"75ee8c7b-b877-4f5c-9b74-08bda044c20c\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:05:30.843325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:30.843258 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:05:31.205597 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:31.205439 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt"] Oct 30 18:05:31.210984 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:05:31.210870 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ee8c7b_b877_4f5c_9b74_08bda044c20c.slice/crio-bf5269208fbdefdc13e4a2fc01254205f537383717f140ff0ae59719c51fe357 WatchSource:0}: Error finding container bf5269208fbdefdc13e4a2fc01254205f537383717f140ff0ae59719c51fe357: Status 404 returned error can't find the container with id bf5269208fbdefdc13e4a2fc01254205f537383717f140ff0ae59719c51fe357 Oct 30 18:05:31.375758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:31.375664 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" event={"ID":"75ee8c7b-b877-4f5c-9b74-08bda044c20c","Type":"ContainerStarted","Data":"bf5269208fbdefdc13e4a2fc01254205f537383717f140ff0ae59719c51fe357"} Oct 30 18:05:31.443065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:31.442862 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:32.257584 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:32.257426 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:32.677317 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:32.677254 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:32.811360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:32.811277 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:33.045487 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:33.045301 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:34.403587 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:34.403505 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" event={"ID":"75ee8c7b-b877-4f5c-9b74-08bda044c20c","Type":"ContainerStarted","Data":"f4e1b6188655e3d4ac69696c8a0deb0ee70d110880c402b53cb1fcf2e5c7aff1"} Oct 30 18:05:34.404460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:34.403678 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:05:34.469303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:34.469065 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" podStartSLOduration=1.516476721 podStartE2EDuration="4.469039613s" totalImagesPullingTime="2.952562892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:34.467644109 +0000 UTC m=+228.856172217" watchObservedRunningTime="2026-10-30 18:05:34.469039613 +0000 UTC m=+228.857567730" Oct 30 18:05:37.346859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:37.346801 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" Oct 30 18:05:38.635186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.635103 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7"] Oct 30 18:05:38.639631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.639588 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:38.657143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.657081 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7"] Oct 30 18:05:38.667277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.667215 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-726w6\"" Oct 30 18:05:38.723076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.723001 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccq6\" (UniqueName: \"kubernetes.io/projected/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-kube-api-access-gccq6\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f2cf7\" (UID: \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:38.723443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.723166 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f2cf7\" (UID: \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:38.824358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.824293 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccq6\" (UniqueName: \"kubernetes.io/projected/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-kube-api-access-gccq6\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f2cf7\" (UID: \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:38.824709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.824384 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f2cf7\" (UID: \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:38.825212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.825166 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f2cf7\" (UID: \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:38.836005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.835923 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:38.853193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.853135 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccq6\" (UniqueName: \"kubernetes.io/projected/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-kube-api-access-gccq6\") pod \"kuadrant-operator-controller-manager-cf8469d7b-f2cf7\" (UID: \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:38.963408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.963230 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:38.995825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:38.995754 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:39.332123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:39.331904 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7"] Oct 30 18:05:39.347134 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:05:39.347075 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb27b0b8_7f6c_4a8f_92e5_458f05217b28.slice/crio-3cfa49042e34ca90caf7c4878d0bed3105ae0744920c92735ca3fc527dfe03d5 WatchSource:0}: Error finding container 3cfa49042e34ca90caf7c4878d0bed3105ae0744920c92735ca3fc527dfe03d5: Status 404 returned error can't find the container with id 3cfa49042e34ca90caf7c4878d0bed3105ae0744920c92735ca3fc527dfe03d5 Oct 30 18:05:39.441109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:39.441044 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" event={"ID":"eb27b0b8-7f6c-4a8f-92e5-458f05217b28","Type":"ContainerStarted","Data":"3cfa49042e34ca90caf7c4878d0bed3105ae0744920c92735ca3fc527dfe03d5"} Oct 30 18:05:39.930142 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:39.930074 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:40.579136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:40.579063 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:42.247201 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:42.247131 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:43.440143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:43.440069 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:43.788182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:43.788001 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:43.951273 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:43.951201 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r8j74"] Oct 30 18:05:43.956062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:43.956014 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" Oct 30 18:05:43.979848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:43.979774 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r8j74"] Oct 30 18:05:43.989737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:43.989672 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jwv4k\"" Oct 30 18:05:44.090521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:44.090339 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75j8\" (UniqueName: \"kubernetes.io/projected/54747d07-9037-4c21-be5c-c554f0b21b5d-kube-api-access-t75j8\") pod \"authorino-operator-7b7bb4bc77-r8j74\" (UID: \"54747d07-9037-4c21-be5c-c554f0b21b5d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" Oct 30 18:05:44.191284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:44.191204 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75j8\" (UniqueName: \"kubernetes.io/projected/54747d07-9037-4c21-be5c-c554f0b21b5d-kube-api-access-t75j8\") pod \"authorino-operator-7b7bb4bc77-r8j74\" (UID: \"54747d07-9037-4c21-be5c-c554f0b21b5d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" Oct 30 18:05:44.218073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:44.218002 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75j8\" (UniqueName: \"kubernetes.io/projected/54747d07-9037-4c21-be5c-c554f0b21b5d-kube-api-access-t75j8\") pod \"authorino-operator-7b7bb4bc77-r8j74\" (UID: \"54747d07-9037-4c21-be5c-c554f0b21b5d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" Oct 30 18:05:44.276744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:44.276663 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" Oct 30 18:05:44.487724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:44.487648 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" event={"ID":"eb27b0b8-7f6c-4a8f-92e5-458f05217b28","Type":"ContainerStarted","Data":"2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e"} Oct 30 18:05:44.488558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:44.487824 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:44.557874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:44.557756 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" podStartSLOduration=2.531989396 podStartE2EDuration="6.557721663s" totalImagesPullingTime="4.025732267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:44.55490092 +0000 UTC m=+238.943429038" watchObservedRunningTime="2026-10-30 18:05:44.557721663 +0000 UTC m=+238.946249780" Oct 30 18:05:44.628611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:44.628561 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-r8j74"] Oct 30 18:05:44.633027 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:05:44.632957 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54747d07_9037_4c21_be5c_c554f0b21b5d.slice/crio-2845f0406af3e52a517905d8b4ddb3247766eb8aec1d61121e449482a2b20e64 WatchSource:0}: Error finding container 2845f0406af3e52a517905d8b4ddb3247766eb8aec1d61121e449482a2b20e64: Status 404 returned error can't find the container with id 2845f0406af3e52a517905d8b4ddb3247766eb8aec1d61121e449482a2b20e64 Oct 30 18:05:45.248332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:45.248264 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:45.415713 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:45.415635 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:05:45.501792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:45.501503 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" event={"ID":"54747d07-9037-4c21-be5c-c554f0b21b5d","Type":"ContainerStarted","Data":"2845f0406af3e52a517905d8b4ddb3247766eb8aec1d61121e449482a2b20e64"} Oct 30 18:05:45.698536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:45.698447 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:46.513403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.513320 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" event={"ID":"54747d07-9037-4c21-be5c-c554f0b21b5d","Type":"ContainerStarted","Data":"7ec86112982d82302166740516aa060d1050fd2e5d2b1ec38ed933bac3a137b0"} Oct 30 18:05:46.514091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.513705 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" Oct 30 18:05:46.539141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.539055 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn"] Oct 30 18:05:46.544097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.544041 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" Oct 30 18:05:46.559271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.559200 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn"] Oct 30 18:05:46.573742 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.573684 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:05:46.574143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.573782 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 18:05:46.574143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.573796 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-t6khc\"" Oct 30 18:05:46.577358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.577236 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" podStartSLOduration=1.884111729 podStartE2EDuration="3.577201643s" totalImagesPullingTime="1.693089914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:46.575514905 +0000 UTC m=+240.964043023" watchObservedRunningTime="2026-10-30 18:05:46.577201643 +0000 UTC m=+240.965729760" Oct 30 18:05:46.637547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.637475 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3fc2f22-610b-4e62-b8a5-0a52070c7847-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn\" (UID: \"a3fc2f22-610b-4e62-b8a5-0a52070c7847\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" Oct 30 18:05:46.637890 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.637644 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gpz\" (UniqueName: \"kubernetes.io/projected/a3fc2f22-610b-4e62-b8a5-0a52070c7847-kube-api-access-m2gpz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn\" (UID: \"a3fc2f22-610b-4e62-b8a5-0a52070c7847\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" Oct 30 18:05:46.739437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.739371 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3fc2f22-610b-4e62-b8a5-0a52070c7847-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn\" (UID: \"a3fc2f22-610b-4e62-b8a5-0a52070c7847\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" Oct 30 18:05:46.739437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.739445 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gpz\" (UniqueName: \"kubernetes.io/projected/a3fc2f22-610b-4e62-b8a5-0a52070c7847-kube-api-access-m2gpz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn\" (UID: \"a3fc2f22-610b-4e62-b8a5-0a52070c7847\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" Oct 30 18:05:46.740123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.740080 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3fc2f22-610b-4e62-b8a5-0a52070c7847-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn\" (UID: \"a3fc2f22-610b-4e62-b8a5-0a52070c7847\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" Oct 30 18:05:46.769903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.769857 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gpz\" (UniqueName: \"kubernetes.io/projected/a3fc2f22-610b-4e62-b8a5-0a52070c7847-kube-api-access-m2gpz\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn\" (UID: \"a3fc2f22-610b-4e62-b8a5-0a52070c7847\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" Oct 30 18:05:46.865989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:46.865874 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" Oct 30 18:05:47.235510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:47.235463 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn"] Oct 30 18:05:47.242105 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:05:47.242038 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fc2f22_610b_4e62_b8a5_0a52070c7847.slice/crio-c910f88a8c1b9edee30a249b6d114ef897d94e708f3e559d5944ebd06a05209d WatchSource:0}: Error finding container c910f88a8c1b9edee30a249b6d114ef897d94e708f3e559d5944ebd06a05209d: Status 404 returned error can't find the container with id c910f88a8c1b9edee30a249b6d114ef897d94e708f3e559d5944ebd06a05209d Oct 30 18:05:47.525271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:47.525080 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" event={"ID":"a3fc2f22-610b-4e62-b8a5-0a52070c7847","Type":"ContainerStarted","Data":"c910f88a8c1b9edee30a249b6d114ef897d94e708f3e559d5944ebd06a05209d"} Oct 30 18:05:47.611913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:47.611849 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:47.746560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:47.746497 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:47.894439 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:47.894328 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:48.064026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.063949 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:48.260746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.260574 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5"] Oct 30 18:05:48.265591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.265516 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:05:48.289647 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.288528 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4865w\"" Oct 30 18:05:48.290015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.289841 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5"] Oct 30 18:05:48.360719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.360503 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ggq\" (UniqueName: \"kubernetes.io/projected/09728775-6be0-45a5-bfb9-480ff7a90759-kube-api-access-p7ggq\") pod \"limitador-operator-controller-manager-5778bb4449-mndc5\" (UID: \"09728775-6be0-45a5-bfb9-480ff7a90759\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:05:48.461882 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.461787 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ggq\" (UniqueName: \"kubernetes.io/projected/09728775-6be0-45a5-bfb9-480ff7a90759-kube-api-access-p7ggq\") pod \"limitador-operator-controller-manager-5778bb4449-mndc5\" (UID: \"09728775-6be0-45a5-bfb9-480ff7a90759\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:05:48.488401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.488332 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ggq\" (UniqueName: \"kubernetes.io/projected/09728775-6be0-45a5-bfb9-480ff7a90759-kube-api-access-p7ggq\") pod \"limitador-operator-controller-manager-5778bb4449-mndc5\" (UID: \"09728775-6be0-45a5-bfb9-480ff7a90759\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:05:48.592961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.592796 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:05:48.964839 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:48.964772 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5"] Oct 30 18:05:49.011635 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:05:49.011566 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09728775_6be0_45a5_bfb9_480ff7a90759.slice/crio-bc49f8b3a1a15de8342e6441b2f2fbbd49cf22bf3062c0cc6d5c80ba31299886 WatchSource:0}: Error finding container bc49f8b3a1a15de8342e6441b2f2fbbd49cf22bf3062c0cc6d5c80ba31299886: Status 404 returned error can't find the container with id bc49f8b3a1a15de8342e6441b2f2fbbd49cf22bf3062c0cc6d5c80ba31299886 Oct 30 18:05:49.547402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:49.547326 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" event={"ID":"a3fc2f22-610b-4e62-b8a5-0a52070c7847","Type":"ContainerStarted","Data":"65f95b6191b231f030a19e051db2dd56fc82b3bb937fd4db2df70694f57e871a"} Oct 30 18:05:49.551170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:49.551095 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" event={"ID":"09728775-6be0-45a5-bfb9-480ff7a90759","Type":"ContainerStarted","Data":"bc49f8b3a1a15de8342e6441b2f2fbbd49cf22bf3062c0cc6d5c80ba31299886"} Oct 30 18:05:49.630771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:49.630628 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-bm2bn" podStartSLOduration=1.856891334 podStartE2EDuration="3.630592238s" totalImagesPullingTime="1.773700904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:49.616283988 +0000 UTC m=+244.004812100" watchObservedRunningTime="2026-10-30 18:05:49.630592238 +0000 UTC m=+244.019120356" Oct 30 18:05:49.942744 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:05:49.942666 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:05:50.562722 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:50.562653 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" event={"ID":"09728775-6be0-45a5-bfb9-480ff7a90759","Type":"ContainerStarted","Data":"5d3918195112a63cf2e8bbe905f64a986bd507845a3025275dda8bfabb3e8d27"} Oct 30 18:05:50.563164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:50.563043 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:05:50.648801 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:50.648676 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" podStartSLOduration=1.323308775 podStartE2EDuration="2.648645148s" totalImagesPullingTime="1.325336373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:50.647416393 +0000 UTC m=+245.035944510" watchObservedRunningTime="2026-10-30 18:05:50.648645148 +0000 UTC m=+245.037173267" Oct 30 18:05:50.766652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:50.766579 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:50.938691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:50.938508 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:51.208666 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:51.208504 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:51.401367 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:51.401313 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:51.540051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:51.539848 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:51.663053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:51.662993 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:53.559479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:53.559410 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:53.715115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:53.715058 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:53.872621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:53.872559 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:54.182499 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.182300 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:54.383384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.383321 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:54.605183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.605006 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd"] Oct 30 18:05:54.611145 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.611089 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.629313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.629259 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gpm9w\"" Oct 30 18:05:54.643189 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.643115 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd"] Oct 30 18:05:54.703378 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.703285 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:54.755297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755195 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.755733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755334 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.755733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755415 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.755733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755440 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.755733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755496 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.755733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755532 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.755733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755585 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.755733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755611 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.755733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.755659 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9h6\" (UniqueName: \"kubernetes.io/projected/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-kube-api-access-js9h6\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.837796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.837696 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:54.857029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.856804 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9h6\" (UniqueName: \"kubernetes.io/projected/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-kube-api-access-js9h6\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.856886 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857011 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857089 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857119 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857191 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857270 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857317 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857348 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857759 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857720 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857818 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.857913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.857890 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.858125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.858090 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.858454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.858424 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.861485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.861415 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.861735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.861608 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.880426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.880347 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.880426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.880384 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9h6\" (UniqueName: \"kubernetes.io/projected/a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7-kube-api-access-js9h6\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-xvbrd\" (UID: \"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:54.936373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:54.936273 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:55.306541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.306476 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd"] Oct 30 18:05:55.309532 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:05:55.309477 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda246d0ff_5e96_46e0_9e23_e3ebdf7b95d7.slice/crio-c426cc8d5ee52aee258a515f7b049d115c6500b5fd93bb9e2a25c2455402c4b0 WatchSource:0}: Error finding container c426cc8d5ee52aee258a515f7b049d115c6500b5fd93bb9e2a25c2455402c4b0: Status 404 returned error can't find the container with id c426cc8d5ee52aee258a515f7b049d115c6500b5fd93bb9e2a25c2455402c4b0 Oct 30 18:05:55.313647 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.313572 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:05:55.313869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.313728 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:05:55.313869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.313773 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:05:55.506653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.506579 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:05:55.606875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.605906 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" event={"ID":"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7","Type":"ContainerStarted","Data":"1a67226d11ceefadb5843692e5f571f2fddb47d8a7975cfb911e83b392911b0d"} Oct 30 18:05:55.606875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.606071 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" event={"ID":"a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7","Type":"ContainerStarted","Data":"c426cc8d5ee52aee258a515f7b049d115c6500b5fd93bb9e2a25c2455402c4b0"} Oct 30 18:05:55.672103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.671956 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" podStartSLOduration=1.671903581 podStartE2EDuration="1.671903581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:55.668678038 +0000 UTC m=+250.057206155" watchObservedRunningTime="2026-10-30 18:05:55.671903581 +0000 UTC m=+250.060431700" Oct 30 18:05:55.937322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.937225 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:55.967053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.966986 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g"] Oct 30 18:05:55.977025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:55.976969 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:56.016844 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.015246 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g"] Oct 30 18:05:56.025303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.025237 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:56.071486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.071414 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4c541a-41f4-4af4-a006-3ce2155611bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x262g\" (UID: \"bd4c541a-41f4-4af4-a006-3ce2155611bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:56.071896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.071563 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltdg\" (UniqueName: \"kubernetes.io/projected/bd4c541a-41f4-4af4-a006-3ce2155611bd-kube-api-access-vltdg\") pod \"kuadrant-operator-controller-manager-678d87766f-x262g\" (UID: \"bd4c541a-41f4-4af4-a006-3ce2155611bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:56.173286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.173203 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4c541a-41f4-4af4-a006-3ce2155611bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x262g\" (UID: \"bd4c541a-41f4-4af4-a006-3ce2155611bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:56.173649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.173315 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltdg\" (UniqueName: \"kubernetes.io/projected/bd4c541a-41f4-4af4-a006-3ce2155611bd-kube-api-access-vltdg\") pod \"kuadrant-operator-controller-manager-678d87766f-x262g\" (UID: \"bd4c541a-41f4-4af4-a006-3ce2155611bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:56.173926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.173889 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4c541a-41f4-4af4-a006-3ce2155611bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x262g\" (UID: \"bd4c541a-41f4-4af4-a006-3ce2155611bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:56.196564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.196416 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltdg\" (UniqueName: \"kubernetes.io/projected/bd4c541a-41f4-4af4-a006-3ce2155611bd-kube-api-access-vltdg\") pod \"kuadrant-operator-controller-manager-678d87766f-x262g\" (UID: \"bd4c541a-41f4-4af4-a006-3ce2155611bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:56.300333 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.300260 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:56.465606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.465447 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:56.608911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.608839 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:56.711301 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.711246 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g"] Oct 30 18:05:56.715490 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:05:56.715435 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4c541a_41f4_4af4_a006_3ce2155611bd.slice/crio-37c9025d584cc068e457a092796f4827d6f53e88c4f222226b12292b4db79cfc WatchSource:0}: Error finding container 37c9025d584cc068e457a092796f4827d6f53e88c4f222226b12292b4db79cfc: Status 404 returned error can't find the container with id 37c9025d584cc068e457a092796f4827d6f53e88c4f222226b12292b4db79cfc Oct 30 18:05:56.948017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:56.947854 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:57.053134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.052900 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:57.156889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.156797 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:57.255161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.255080 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:57.529903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.529849 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" Oct 30 18:05:57.627602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.627525 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" event={"ID":"bd4c541a-41f4-4af4-a006-3ce2155611bd","Type":"ContainerStarted","Data":"695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a"} Oct 30 18:05:57.628466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.627701 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:57.628466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.628082 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:05:57.628466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.628137 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" event={"ID":"bd4c541a-41f4-4af4-a006-3ce2155611bd","Type":"ContainerStarted","Data":"37c9025d584cc068e457a092796f4827d6f53e88c4f222226b12292b4db79cfc"} Oct 30 18:05:57.630875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.630813 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-xvbrd" Oct 30 18:05:57.732601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.732497 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:57.776379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.775651 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podStartSLOduration=2.775602497 podStartE2EDuration="2.775602497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:57.752792513 +0000 UTC m=+252.141320630" watchObservedRunningTime="2026-10-30 18:05:57.775602497 +0000 UTC m=+252.164130615" Oct 30 18:05:57.928032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:05:57.927956 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:01.067507 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:01.067429 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:01.575066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:01.575002 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:06:03.767166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:03.767072 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:03.911581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:03.909186 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:04.334988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:04.334903 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:05.861306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:05.861236 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:05.960155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:05.960087 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:08.640013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:08.639929 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:06:08.750063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:08.750011 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7"] Oct 30 18:06:08.750452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:08.750413 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" podUID="eb27b0b8-7f6c-4a8f-92e5-458f05217b28" containerName="manager" containerID="cri-o://2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e" gracePeriod=10 Oct 30 18:06:08.902984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:08.902819 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:09.022519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.022452 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:09.083341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.083268 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:06:09.150751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.150674 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-extensions-socket-volume\") pod \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\" (UID: \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\") " Oct 30 18:06:09.151125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.150805 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-kube-api-access-gccq6\" (UniqueName: \"kubernetes.io/projected/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-kube-api-access-gccq6\") pod \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\" (UID: \"eb27b0b8-7f6c-4a8f-92e5-458f05217b28\") " Oct 30 18:06:09.151322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.151285 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-extensions-socket-volume" pod "eb27b0b8-7f6c-4a8f-92e5-458f05217b28" (UID: "eb27b0b8-7f6c-4a8f-92e5-458f05217b28"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:09.154845 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.154658 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-kube-api-access-gccq6" pod "eb27b0b8-7f6c-4a8f-92e5-458f05217b28" (UID: "eb27b0b8-7f6c-4a8f-92e5-458f05217b28"). InnerVolumeSpecName "kube-api-access-gccq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:09.252799 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.252714 5624 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-extensions-socket-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:06:09.252799 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.252778 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gccq6\" (UniqueName: \"kubernetes.io/projected/eb27b0b8-7f6c-4a8f-92e5-458f05217b28-kube-api-access-gccq6\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:06:09.741981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.741882 5624 generic.go:356] "Generic (PLEG): container finished" podID="eb27b0b8-7f6c-4a8f-92e5-458f05217b28" containerID="2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e" exitCode=0 Oct 30 18:06:09.742819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.741988 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" event={"ID":"eb27b0b8-7f6c-4a8f-92e5-458f05217b28","Type":"ContainerDied","Data":"2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e"} Oct 30 18:06:09.742819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.742014 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" Oct 30 18:06:09.742819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.742074 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7" event={"ID":"eb27b0b8-7f6c-4a8f-92e5-458f05217b28","Type":"ContainerDied","Data":"3cfa49042e34ca90caf7c4878d0bed3105ae0744920c92735ca3fc527dfe03d5"} Oct 30 18:06:09.742819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.742107 5624 scope.go:122] "RemoveContainer" containerID="2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e" Oct 30 18:06:09.759121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.758863 5624 scope.go:122] "RemoveContainer" containerID="2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e" Oct 30 18:06:09.759519 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:06:09.759488 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e\": container with ID starting with 2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e not found: ID does not exist" containerID="2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e" Oct 30 18:06:09.759611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.759539 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e"} err="failed to get container status \"2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e\": rpc error: code = NotFound desc = could not find container \"2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e\": container with ID starting with 2faf728b989bb735d9f1f5167a57a92b52ac52fd1556bd59e9f451fa2140759e not found: ID does not exist" Oct 30 18:06:09.819340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.819267 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7"] Oct 30 18:06:09.837424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:09.837354 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-f2cf7"] Oct 30 18:06:11.486021 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:11.485953 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:11.612612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:11.612540 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:11.792135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:11.791985 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb27b0b8-7f6c-4a8f-92e5-458f05217b28" path="/var/lib/kubelet/pods/eb27b0b8-7f6c-4a8f-92e5-458f05217b28/volumes" Oct 30 18:06:15.620377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:15.620320 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:15.719636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:15.719554 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:16.380856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.380668 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9kckb"] Oct 30 18:06:16.381672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.381623 5624 cpu_manager.go:414] "containerMap: removing container" podUID="eb27b0b8-7f6c-4a8f-92e5-458f05217b28" containerName="manager" Oct 30 18:06:16.381672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.381675 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb27b0b8-7f6c-4a8f-92e5-458f05217b28" containerName="manager" Oct 30 18:06:16.381906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.381821 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb27b0b8-7f6c-4a8f-92e5-458f05217b28" containerName="manager" Oct 30 18:06:16.386195 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.386128 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9kckb" Oct 30 18:06:16.403462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.403396 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9kckb"] Oct 30 18:06:16.412113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.412051 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b5sg4\"" Oct 30 18:06:16.444686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.444611 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:16.457138 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.457059 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vht6j\" (UniqueName: \"kubernetes.io/projected/050ee0cf-faa1-40f3-98ee-a9c2c04122b7-kube-api-access-vht6j\") pod \"authorino-57f7f8659f-9kckb\" (UID: \"050ee0cf-faa1-40f3-98ee-a9c2c04122b7\") " pod="kuadrant-system/authorino-57f7f8659f-9kckb" Oct 30 18:06:16.545489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.545415 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:16.558856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.558766 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vht6j\" (UniqueName: \"kubernetes.io/projected/050ee0cf-faa1-40f3-98ee-a9c2c04122b7-kube-api-access-vht6j\") pod \"authorino-57f7f8659f-9kckb\" (UID: \"050ee0cf-faa1-40f3-98ee-a9c2c04122b7\") " pod="kuadrant-system/authorino-57f7f8659f-9kckb" Oct 30 18:06:16.580905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.580827 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vht6j\" (UniqueName: \"kubernetes.io/projected/050ee0cf-faa1-40f3-98ee-a9c2c04122b7-kube-api-access-vht6j\") pod \"authorino-57f7f8659f-9kckb\" (UID: \"050ee0cf-faa1-40f3-98ee-a9c2c04122b7\") " pod="kuadrant-system/authorino-57f7f8659f-9kckb" Oct 30 18:06:16.718213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.718046 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9kckb" Oct 30 18:06:16.847680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.847592 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:16.939213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:16.939149 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:17.051626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:17.051570 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9kckb"] Oct 30 18:06:17.055737 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:06:17.055678 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod050ee0cf_faa1_40f3_98ee_a9c2c04122b7.slice/crio-f2eae96cbf491a56ee4cb0ffeafb6002878667f422add09fc207d56937ec0877 WatchSource:0}: Error finding container f2eae96cbf491a56ee4cb0ffeafb6002878667f422add09fc207d56937ec0877: Status 404 returned error can't find the container with id f2eae96cbf491a56ee4cb0ffeafb6002878667f422add09fc207d56937ec0877 Oct 30 18:06:17.259734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:17.259676 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:17.371408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:17.371243 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:17.645159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:17.645094 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:17.743311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:17.743246 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:17.821532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:17.821458 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9kckb" event={"ID":"050ee0cf-faa1-40f3-98ee-a9c2c04122b7","Type":"ContainerStarted","Data":"f2eae96cbf491a56ee4cb0ffeafb6002878667f422add09fc207d56937ec0877"} Oct 30 18:06:19.843620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:19.843445 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9kckb" event={"ID":"050ee0cf-faa1-40f3-98ee-a9c2c04122b7","Type":"ContainerStarted","Data":"fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d"} Oct 30 18:06:19.885958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:19.885809 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-9kckb" podStartSLOduration=2.176582775 podStartE2EDuration="3.885774551s" totalImagesPullingTime="1.709191776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:19.883632814 +0000 UTC m=+274.272160931" watchObservedRunningTime="2026-10-30 18:06:19.885774551 +0000 UTC m=+274.274303358" Oct 30 18:06:19.937834 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:06:19.937751 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:06:20.726715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:20.726643 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:25.697853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:25.697708 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:25.811197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:25.811123 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:26.718621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:26.718528 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mwclg"] Oct 30 18:06:26.725393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:26.725330 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" Oct 30 18:06:26.741346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:26.741281 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mwclg"] Oct 30 18:06:26.770364 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:26.770300 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:26.891768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:26.891697 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hfmz\" (UniqueName: \"kubernetes.io/projected/d2563a61-e976-47ab-8d03-1772aa98150f-kube-api-access-8hfmz\") pod \"authorino-56c6c9ddd5-mwclg\" (UID: \"d2563a61-e976-47ab-8d03-1772aa98150f\") " pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" Oct 30 18:06:26.992805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:26.992590 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hfmz\" (UniqueName: \"kubernetes.io/projected/d2563a61-e976-47ab-8d03-1772aa98150f-kube-api-access-8hfmz\") pod \"authorino-56c6c9ddd5-mwclg\" (UID: \"d2563a61-e976-47ab-8d03-1772aa98150f\") " pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" Oct 30 18:06:27.016296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.016219 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hfmz\" (UniqueName: \"kubernetes.io/projected/d2563a61-e976-47ab-8d03-1772aa98150f-kube-api-access-8hfmz\") pod \"authorino-56c6c9ddd5-mwclg\" (UID: \"d2563a61-e976-47ab-8d03-1772aa98150f\") " pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" Oct 30 18:06:27.049721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.049657 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" Oct 30 18:06:27.385170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.385105 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mwclg"] Oct 30 18:06:27.388879 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:06:27.388818 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2563a61_e976_47ab_8d03_1772aa98150f.slice/crio-80b1e8747a7b37414ddd3bc69cc76fa8a908bac8cf260e124111585d1356dfc5 WatchSource:0}: Error finding container 80b1e8747a7b37414ddd3bc69cc76fa8a908bac8cf260e124111585d1356dfc5: Status 404 returned error can't find the container with id 80b1e8747a7b37414ddd3bc69cc76fa8a908bac8cf260e124111585d1356dfc5 Oct 30 18:06:27.462715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.462612 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:06:27.470534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.470463 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:27.485664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.485617 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:06:27.495235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.495170 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 18:06:27.495235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.495207 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dcw6s\"" Oct 30 18:06:27.555638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.555567 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:06:27.599026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.598895 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c4422bc-c3e1-48f1-99a8-c62260e1d25c-config-file\") pod \"limitador-limitador-7c9b87d574-2hj6q\" (UID: \"5c4422bc-c3e1-48f1-99a8-c62260e1d25c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:27.599361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.599114 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qlzv\" (UniqueName: \"kubernetes.io/projected/5c4422bc-c3e1-48f1-99a8-c62260e1d25c-kube-api-access-4qlzv\") pod \"limitador-limitador-7c9b87d574-2hj6q\" (UID: \"5c4422bc-c3e1-48f1-99a8-c62260e1d25c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:27.701172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.700925 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c4422bc-c3e1-48f1-99a8-c62260e1d25c-config-file\") pod \"limitador-limitador-7c9b87d574-2hj6q\" (UID: \"5c4422bc-c3e1-48f1-99a8-c62260e1d25c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:27.701491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.701169 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qlzv\" (UniqueName: \"kubernetes.io/projected/5c4422bc-c3e1-48f1-99a8-c62260e1d25c-kube-api-access-4qlzv\") pod \"limitador-limitador-7c9b87d574-2hj6q\" (UID: \"5c4422bc-c3e1-48f1-99a8-c62260e1d25c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:27.702376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.702334 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5c4422bc-c3e1-48f1-99a8-c62260e1d25c-config-file\") pod \"limitador-limitador-7c9b87d574-2hj6q\" (UID: \"5c4422bc-c3e1-48f1-99a8-c62260e1d25c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:27.728108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.728042 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qlzv\" (UniqueName: \"kubernetes.io/projected/5c4422bc-c3e1-48f1-99a8-c62260e1d25c-kube-api-access-4qlzv\") pod \"limitador-limitador-7c9b87d574-2hj6q\" (UID: \"5c4422bc-c3e1-48f1-99a8-c62260e1d25c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:27.797081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.797028 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:27.919512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.919362 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" event={"ID":"d2563a61-e976-47ab-8d03-1772aa98150f","Type":"ContainerStarted","Data":"38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784"} Oct 30 18:06:27.919512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.919436 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" event={"ID":"d2563a61-e976-47ab-8d03-1772aa98150f","Type":"ContainerStarted","Data":"80b1e8747a7b37414ddd3bc69cc76fa8a908bac8cf260e124111585d1356dfc5"} Oct 30 18:06:27.962152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.961907 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" podStartSLOduration=1.576257704 podStartE2EDuration="1.961875738s" totalImagesPullingTime="385.618034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:27.959245896 +0000 UTC m=+282.347774014" watchObservedRunningTime="2026-10-30 18:06:27.961875738 +0000 UTC m=+282.350403854" Oct 30 18:06:27.989582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.989487 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9kckb"] Oct 30 18:06:27.990066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:27.990026 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-9kckb" podUID="050ee0cf-faa1-40f3-98ee-a9c2c04122b7" containerName="authorino" containerID="cri-o://fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d" gracePeriod=30 Oct 30 18:06:28.155044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.154999 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:06:28.160641 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:06:28.160557 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4422bc_c3e1_48f1_99a8_c62260e1d25c.slice/crio-4d93651f713f36d5e135782fd59f11c3eb73831906c9ff36861a7929edaf8e58 WatchSource:0}: Error finding container 4d93651f713f36d5e135782fd59f11c3eb73831906c9ff36861a7929edaf8e58: Status 404 returned error can't find the container with id 4d93651f713f36d5e135782fd59f11c3eb73831906c9ff36861a7929edaf8e58 Oct 30 18:06:28.308990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.308899 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9kckb" Oct 30 18:06:28.416439 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.416365 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050ee0cf-faa1-40f3-98ee-a9c2c04122b7-kube-api-access-vht6j\" (UniqueName: \"kubernetes.io/projected/050ee0cf-faa1-40f3-98ee-a9c2c04122b7-kube-api-access-vht6j\") pod \"050ee0cf-faa1-40f3-98ee-a9c2c04122b7\" (UID: \"050ee0cf-faa1-40f3-98ee-a9c2c04122b7\") " Oct 30 18:06:28.420232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.420150 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050ee0cf-faa1-40f3-98ee-a9c2c04122b7-kube-api-access-vht6j" pod "050ee0cf-faa1-40f3-98ee-a9c2c04122b7" (UID: "050ee0cf-faa1-40f3-98ee-a9c2c04122b7"). InnerVolumeSpecName "kube-api-access-vht6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:28.517720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.517653 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vht6j\" (UniqueName: \"kubernetes.io/projected/050ee0cf-faa1-40f3-98ee-a9c2c04122b7-kube-api-access-vht6j\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:06:28.692828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.692729 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:28.815829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.815751 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:28.932432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.932358 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" event={"ID":"5c4422bc-c3e1-48f1-99a8-c62260e1d25c","Type":"ContainerStarted","Data":"4d93651f713f36d5e135782fd59f11c3eb73831906c9ff36861a7929edaf8e58"} Oct 30 18:06:28.935811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.935723 5624 generic.go:356] "Generic (PLEG): container finished" podID="050ee0cf-faa1-40f3-98ee-a9c2c04122b7" containerID="fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d" exitCode=0 Oct 30 18:06:28.936171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.935846 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-9kckb" Oct 30 18:06:28.936171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.935834 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9kckb" event={"ID":"050ee0cf-faa1-40f3-98ee-a9c2c04122b7","Type":"ContainerDied","Data":"fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d"} Oct 30 18:06:28.936171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.936005 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-9kckb" event={"ID":"050ee0cf-faa1-40f3-98ee-a9c2c04122b7","Type":"ContainerDied","Data":"f2eae96cbf491a56ee4cb0ffeafb6002878667f422add09fc207d56937ec0877"} Oct 30 18:06:28.936171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.936030 5624 scope.go:122] "RemoveContainer" containerID="fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d" Oct 30 18:06:28.955121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.955074 5624 scope.go:122] "RemoveContainer" containerID="fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d" Oct 30 18:06:28.955897 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:06:28.955837 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d\": container with ID starting with fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d not found: ID does not exist" containerID="fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d" Oct 30 18:06:28.956107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.955918 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d"} err="failed to get container status \"fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d\": rpc error: code = NotFound desc = could not find container \"fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d\": container with ID starting with fb544abb9846806bf306f0db9cb5abee8a0dd69825cfcf49543c3e8cae79bd0d not found: ID does not exist" Oct 30 18:06:28.992305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.992236 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9kckb"] Oct 30 18:06:28.997065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:28.997003 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-9kckb"] Oct 30 18:06:29.792441 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:29.792349 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050ee0cf-faa1-40f3-98ee-a9c2c04122b7" path="/var/lib/kubelet/pods/050ee0cf-faa1-40f3-98ee-a9c2c04122b7/volumes" Oct 30 18:06:29.946662 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:29.946596 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" event={"ID":"5c4422bc-c3e1-48f1-99a8-c62260e1d25c","Type":"ContainerStarted","Data":"3951d54972fd5e6b9b2bf984915fe365ae1baf615bbdfbc3e8d924f1310659ce"} Oct 30 18:06:29.947462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:29.946729 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:29.990109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:29.989994 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" podStartSLOduration=2.053868635 podStartE2EDuration="2.989921578s" totalImagesPullingTime="936.052943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:29.988782277 +0000 UTC m=+284.377310397" watchObservedRunningTime="2026-10-30 18:06:29.989921578 +0000 UTC m=+284.378449697" Oct 30 18:06:39.081984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:39.081887 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:40.959905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:40.959837 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" Oct 30 18:06:44.420521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:44.420440 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:44.515902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:44.515841 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:47.219589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:47.219521 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:47.333916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:47.333843 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:48.025606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:48.025527 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:49.930538 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:06:49.930474 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:06:58.416587 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.416502 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:58.518827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.518733 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:58.615893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.615824 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:58.833114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.832898 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:58.844497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.844412 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-49l6m"] Oct 30 18:06:58.845552 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.845489 5624 cpu_manager.go:414] "containerMap: removing container" podUID="050ee0cf-faa1-40f3-98ee-a9c2c04122b7" containerName="authorino" Oct 30 18:06:58.845552 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.845539 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050ee0cf-faa1-40f3-98ee-a9c2c04122b7" containerName="authorino" Oct 30 18:06:58.845867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.845655 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="050ee0cf-faa1-40f3-98ee-a9c2c04122b7" containerName="authorino" Oct 30 18:06:58.850605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.850542 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:06:58.871089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.870956 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-49l6m"] Oct 30 18:06:58.879827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.879740 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:06:58.879827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.879741 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:06:58.892783 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.892686 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-84jnk"] Oct 30 18:06:58.893466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.893408 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98g7m\" (UniqueName: \"kubernetes.io/projected/0a09133a-00c6-4632-ba5e-b859f8da282e-kube-api-access-98g7m\") pod \"model-serving-api-75bd89fdfc-49l6m\" (UID: \"0a09133a-00c6-4632-ba5e-b859f8da282e\") " pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:06:58.893701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.893478 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a09133a-00c6-4632-ba5e-b859f8da282e-tls-certs\") pod \"model-serving-api-75bd89fdfc-49l6m\" (UID: \"0a09133a-00c6-4632-ba5e-b859f8da282e\") " pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:06:58.898037 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.897902 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:58.900468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.900420 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 18:06:58.901108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.901059 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9ptn6\"" Oct 30 18:06:58.917681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.917622 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-84jnk"] Oct 30 18:06:58.918373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.918335 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jv9hg\"" Oct 30 18:06:58.918373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.918335 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 18:06:58.918621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.918477 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 18:06:58.945708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.945628 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:58.995359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.995274 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5jp\" (UniqueName: \"kubernetes.io/projected/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-kube-api-access-dx5jp\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:58.995760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.995400 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-cert\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:58.995760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.995538 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-metrics-certs\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:58.995760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.995677 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98g7m\" (UniqueName: \"kubernetes.io/projected/0a09133a-00c6-4632-ba5e-b859f8da282e-kube-api-access-98g7m\") pod \"model-serving-api-75bd89fdfc-49l6m\" (UID: \"0a09133a-00c6-4632-ba5e-b859f8da282e\") " pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:06:58.995760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.995720 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a09133a-00c6-4632-ba5e-b859f8da282e-tls-certs\") pod \"model-serving-api-75bd89fdfc-49l6m\" (UID: \"0a09133a-00c6-4632-ba5e-b859f8da282e\") " pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:06:59.000046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:58.999968 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a09133a-00c6-4632-ba5e-b859f8da282e-tls-certs\") pod \"model-serving-api-75bd89fdfc-49l6m\" (UID: \"0a09133a-00c6-4632-ba5e-b859f8da282e\") " pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:06:59.018950 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.018865 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98g7m\" (UniqueName: \"kubernetes.io/projected/0a09133a-00c6-4632-ba5e-b859f8da282e-kube-api-access-98g7m\") pod \"model-serving-api-75bd89fdfc-49l6m\" (UID: \"0a09133a-00c6-4632-ba5e-b859f8da282e\") " pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:06:59.097464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.097259 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5jp\" (UniqueName: \"kubernetes.io/projected/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-kube-api-access-dx5jp\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:59.097464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.097423 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-cert\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:59.097464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.097452 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-metrics-certs\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:59.102078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.102039 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-metrics-certs\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:59.102388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.102181 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-cert\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:59.121923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.121846 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5jp\" (UniqueName: \"kubernetes.io/projected/dc1ecec4-614d-4cbf-978b-7b0fdb293d0f-kube-api-access-dx5jp\") pod \"odh-model-controller-769875db56-84jnk\" (UID: \"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f\") " pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:59.172648 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.172534 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:06:59.223239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.223143 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:06:59.251049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.250981 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:59.516365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.516302 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-49l6m"] Oct 30 18:06:59.519285 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:06:59.519232 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a09133a_00c6_4632_ba5e_b859f8da282e.slice/crio-fee9932c183f77ad6b1dc332e85c9dc784a23fe807ae86edbc11da50c4992b7e WatchSource:0}: Error finding container fee9932c183f77ad6b1dc332e85c9dc784a23fe807ae86edbc11da50c4992b7e: Status 404 returned error can't find the container with id fee9932c183f77ad6b1dc332e85c9dc784a23fe807ae86edbc11da50c4992b7e Oct 30 18:06:59.577629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:06:59.577576 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-84jnk"] Oct 30 18:06:59.580558 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:06:59.580501 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1ecec4_614d_4cbf_978b_7b0fdb293d0f.slice/crio-df2bd57d197d8fcf06597d6ba657946b9daa6319c9d1b85d65d6ca635557c2ce WatchSource:0}: Error finding container df2bd57d197d8fcf06597d6ba657946b9daa6319c9d1b85d65d6ca635557c2ce: Status 404 returned error can't find the container with id df2bd57d197d8fcf06597d6ba657946b9daa6319c9d1b85d65d6ca635557c2ce Oct 30 18:07:00.209351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:00.209265 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" event={"ID":"0a09133a-00c6-4632-ba5e-b859f8da282e","Type":"ContainerStarted","Data":"fee9932c183f77ad6b1dc332e85c9dc784a23fe807ae86edbc11da50c4992b7e"} Oct 30 18:07:00.213641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:00.213479 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-84jnk" event={"ID":"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f","Type":"ContainerStarted","Data":"df2bd57d197d8fcf06597d6ba657946b9daa6319c9d1b85d65d6ca635557c2ce"} Oct 30 18:07:01.178640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:01.178585 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:01.279038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:01.278966 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:01.378706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:01.378634 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:01.482188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:01.482054 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:01.580825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:01.580729 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:01.680216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:01.680138 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:02.239128 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:02.239050 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" event={"ID":"0a09133a-00c6-4632-ba5e-b859f8da282e","Type":"ContainerStarted","Data":"ee7f7183ce7b9d0d66e8c3c2c26065b465ff9778193abcf7c3709993991d7f0d"} Oct 30 18:07:02.239923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:02.239509 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:07:02.288660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:02.288527 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" podStartSLOduration=2.087664864 podStartE2EDuration="4.2884895s" totalImagesPullingTime="2.200824636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:02.286541071 +0000 UTC m=+316.675069190" watchObservedRunningTime="2026-10-30 18:07:02.2884895 +0000 UTC m=+316.677017617" Oct 30 18:07:03.251075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:03.251005 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-84jnk" event={"ID":"dc1ecec4-614d-4cbf-978b-7b0fdb293d0f","Type":"ContainerStarted","Data":"0c138ebc42de436f626169370b3d039870e2c4daf7c8cea5e749fc9061d0f1ab"} Oct 30 18:07:03.251964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:03.251125 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:07:03.297111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:03.297017 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-84jnk" podStartSLOduration=2.05706892 podStartE2EDuration="5.296974936s" totalImagesPullingTime="3.239906016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:03.29454596 +0000 UTC m=+317.683074077" watchObservedRunningTime="2026-10-30 18:07:03.296974936 +0000 UTC m=+317.685503052" Oct 30 18:07:07.092119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.092051 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:07.149681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.149606 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hp2hf"] Oct 30 18:07:07.155045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.154983 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.192811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.192739 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 18:07:07.192811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.192739 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 18:07:07.192811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.192762 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r2cqt\"" Oct 30 18:07:07.193308 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.193066 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 18:07:07.197342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.197264 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hp2hf"] Oct 30 18:07:07.203325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.203256 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmw55\" (UniqueName: \"kubernetes.io/projected/4dd7d71f-f5e5-4e04-9065-caecb25a6379-kube-api-access-zmw55\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.203699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.203520 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-metrics-certs\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.203699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.203646 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-webhook-cert\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.216073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.216021 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:07.305539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.305444 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-metrics-certs\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.306002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.305572 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-webhook-cert\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.306002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.305632 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmw55\" (UniqueName: \"kubernetes.io/projected/4dd7d71f-f5e5-4e04-9065-caecb25a6379-kube-api-access-zmw55\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.310046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.309997 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-webhook-cert\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.310224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.310018 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-metrics-certs\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.317444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.317365 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:07.330332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.330249 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmw55\" (UniqueName: \"kubernetes.io/projected/4dd7d71f-f5e5-4e04-9065-caecb25a6379-kube-api-access-zmw55\") pod \"maas-controller-579cf7c6d-hp2hf\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.426876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.426815 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:07.477127 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.477064 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:07.816440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:07.816350 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hp2hf"] Oct 30 18:07:07.819418 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:07:07.819341 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd7d71f_f5e5_4e04_9065_caecb25a6379.slice/crio-0c6e839bb889e0f22e1da7c5c324b3b47098f10a0d96d77393d89164d7e8014e WatchSource:0}: Error finding container 0c6e839bb889e0f22e1da7c5c324b3b47098f10a0d96d77393d89164d7e8014e: Status 404 returned error can't find the container with id 0c6e839bb889e0f22e1da7c5c324b3b47098f10a0d96d77393d89164d7e8014e Oct 30 18:07:08.304859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:08.304787 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" event={"ID":"4dd7d71f-f5e5-4e04-9065-caecb25a6379","Type":"ContainerStarted","Data":"0c6e839bb889e0f22e1da7c5c324b3b47098f10a0d96d77393d89164d7e8014e"} Oct 30 18:07:10.669882 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:10.669815 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:10.772884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:10.772793 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:10.893824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:10.893730 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:11.336301 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:11.336227 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" event={"ID":"4dd7d71f-f5e5-4e04-9065-caecb25a6379","Type":"ContainerStarted","Data":"152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff"} Oct 30 18:07:11.336654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:11.336361 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:11.397091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:11.396985 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" podStartSLOduration=1.5602230559999999 podStartE2EDuration="4.396908996s" totalImagesPullingTime="2.83668594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:11.395552969 +0000 UTC m=+325.784081087" watchObservedRunningTime="2026-10-30 18:07:11.396908996 +0000 UTC m=+325.785437107" Oct 30 18:07:12.905575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:12.905461 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:13.073853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:13.073783 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:13.257127 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:13.256997 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" Oct 30 18:07:14.263227 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:14.263166 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:07:19.930318 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:07:19.930239 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:07:22.074660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.074455 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hp2hf"] Oct 30 18:07:22.075486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.075010 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" podUID="4dd7d71f-f5e5-4e04-9065-caecb25a6379" containerName="manager" containerID="cri-o://152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff" gracePeriod=30 Oct 30 18:07:22.078145 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.078095 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:22.396059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.395991 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:22.428560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.428492 5624 generic.go:356] "Generic (PLEG): container finished" podID="4dd7d71f-f5e5-4e04-9065-caecb25a6379" containerID="152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff" exitCode=0 Oct 30 18:07:22.428891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.428652 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" event={"ID":"4dd7d71f-f5e5-4e04-9065-caecb25a6379","Type":"ContainerDied","Data":"152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff"} Oct 30 18:07:22.428891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.428671 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" Oct 30 18:07:22.428891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.428782 5624 scope.go:122] "RemoveContainer" containerID="152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff" Oct 30 18:07:22.428891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.428763 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-hp2hf" event={"ID":"4dd7d71f-f5e5-4e04-9065-caecb25a6379","Type":"ContainerDied","Data":"0c6e839bb889e0f22e1da7c5c324b3b47098f10a0d96d77393d89164d7e8014e"} Oct 30 18:07:22.445259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.445223 5624 scope.go:122] "RemoveContainer" containerID="152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff" Oct 30 18:07:22.446005 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:07:22.445912 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff\": container with ID starting with 152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff not found: ID does not exist" containerID="152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff" Oct 30 18:07:22.446213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.446024 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff"} err="failed to get container status \"152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff\": rpc error: code = NotFound desc = could not find container \"152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff\": container with ID starting with 152b244986ec52dbd9b202a74a09e83b2140c60018dbfa736ed4b81e237431ff not found: ID does not exist" Oct 30 18:07:22.506913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.506843 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd7d71f-f5e5-4e04-9065-caecb25a6379-kube-api-access-zmw55\" (UniqueName: \"kubernetes.io/projected/4dd7d71f-f5e5-4e04-9065-caecb25a6379-kube-api-access-zmw55\") pod \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " Oct 30 18:07:22.507310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.506999 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-metrics-certs\") pod \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " Oct 30 18:07:22.507310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.507105 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-webhook-cert\") pod \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\" (UID: \"4dd7d71f-f5e5-4e04-9065-caecb25a6379\") " Oct 30 18:07:22.510737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.510663 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-metrics-certs" pod "4dd7d71f-f5e5-4e04-9065-caecb25a6379" (UID: "4dd7d71f-f5e5-4e04-9065-caecb25a6379"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:22.511065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.510675 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd7d71f-f5e5-4e04-9065-caecb25a6379-kube-api-access-zmw55" pod "4dd7d71f-f5e5-4e04-9065-caecb25a6379" (UID: "4dd7d71f-f5e5-4e04-9065-caecb25a6379"). InnerVolumeSpecName "kube-api-access-zmw55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:22.511065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.510660 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-webhook-cert" pod "4dd7d71f-f5e5-4e04-9065-caecb25a6379" (UID: "4dd7d71f-f5e5-4e04-9065-caecb25a6379"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:22.608996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.608880 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.608996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.609014 5624 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dd7d71f-f5e5-4e04-9065-caecb25a6379-webhook-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.609357 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.609030 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmw55\" (UniqueName: \"kubernetes.io/projected/4dd7d71f-f5e5-4e04-9065-caecb25a6379-kube-api-access-zmw55\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:07:22.781896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.781799 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hp2hf"] Oct 30 18:07:22.786180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:22.786101 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-hp2hf"] Oct 30 18:07:23.792603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:23.792512 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd7d71f-f5e5-4e04-9065-caecb25a6379" path="/var/lib/kubelet/pods/4dd7d71f-f5e5-4e04-9065-caecb25a6379/volumes" Oct 30 18:07:39.206497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:39.206400 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:42.905773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:42.905680 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:43.033116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:43.033064 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:43.174917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:43.174740 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:44.527487 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:44.527403 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:44.641819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:44.641747 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:44.744359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:44.744277 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:45.071451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.071371 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh"] Oct 30 18:07:45.072143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.072105 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4dd7d71f-f5e5-4e04-9065-caecb25a6379" containerName="manager" Oct 30 18:07:45.072143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.072143 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd7d71f-f5e5-4e04-9065-caecb25a6379" containerName="manager" Oct 30 18:07:45.072297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.072239 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd7d71f-f5e5-4e04-9065-caecb25a6379" containerName="manager" Oct 30 18:07:45.084728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.084666 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.095401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.095316 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh"] Oct 30 18:07:45.104928 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.104850 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:07:45.105291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.105069 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:07:45.114376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.114322 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 18:07:45.114376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.114321 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:07:45.114759 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.114470 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:07:45.169776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.169686 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614dc6f6-4f2d-4e15-8219-f67e449f9739-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.170183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.169855 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-metrics-certs\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.170183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.169904 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-maas-api-tls\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.170183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.169984 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glq6m\" (UniqueName: \"kubernetes.io/projected/614dc6f6-4f2d-4e15-8219-f67e449f9739-kube-api-access-glq6m\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.271191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.271117 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614dc6f6-4f2d-4e15-8219-f67e449f9739-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.271514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.271233 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-metrics-certs\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.271514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.271266 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-maas-api-tls\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.271514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.271288 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glq6m\" (UniqueName: \"kubernetes.io/projected/614dc6f6-4f2d-4e15-8219-f67e449f9739-kube-api-access-glq6m\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.272432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.272388 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614dc6f6-4f2d-4e15-8219-f67e449f9739-openshift-service-ca\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.275482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.275416 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-maas-api-tls\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.275735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.275563 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-metrics-certs\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.294436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.294377 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glq6m\" (UniqueName: \"kubernetes.io/projected/614dc6f6-4f2d-4e15-8219-f67e449f9739-kube-api-access-glq6m\") pod \"maas-api-6d669c7fcd-2bgxh\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.410545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.410463 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:45.752275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:45.752180 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh"] Oct 30 18:07:45.753571 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:07:45.753518 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614dc6f6_4f2d_4e15_8219_f67e449f9739.slice/crio-851f5e33e55cb201b8ae93e89b7d4a9993be10dc97187a7e9246c2f76b713651 WatchSource:0}: Error finding container 851f5e33e55cb201b8ae93e89b7d4a9993be10dc97187a7e9246c2f76b713651: Status 404 returned error can't find the container with id 851f5e33e55cb201b8ae93e89b7d4a9993be10dc97187a7e9246c2f76b713651 Oct 30 18:07:46.343166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:46.343081 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:07:46.634057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:46.633980 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" event={"ID":"614dc6f6-4f2d-4e15-8219-f67e449f9739","Type":"ContainerStarted","Data":"851f5e33e55cb201b8ae93e89b7d4a9993be10dc97187a7e9246c2f76b713651"} Oct 30 18:07:48.656084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:48.656023 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" event={"ID":"614dc6f6-4f2d-4e15-8219-f67e449f9739","Type":"ContainerStarted","Data":"71dd491b1b89cb0631e0892155f75828c3d3ccc0e8a146fe6eedfda05a4560ee"} Oct 30 18:07:48.657113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:48.657078 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:48.709710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:48.709559 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" podStartSLOduration=1.494982996 podStartE2EDuration="3.709510184s" totalImagesPullingTime="2.214527188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:48.706452177 +0000 UTC m=+363.094980294" watchObservedRunningTime="2026-10-30 18:07:48.709510184 +0000 UTC m=+363.098038304" Oct 30 18:07:49.957569 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:07:49.957489 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:07:55.724525 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:55.724437 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:07:56.187184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.187113 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:56.429956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.429873 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:56.548816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.548636 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-272wd"] Oct 30 18:07:56.553638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.553574 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-272wd" Oct 30 18:07:56.572378 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.572311 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-272wd"] Oct 30 18:07:56.632869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.632772 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhx8\" (UniqueName: \"kubernetes.io/projected/63f70bd9-cc95-455b-98ab-7d313d37f1a8-kube-api-access-wwhx8\") pod \"authorino-6f6944bcdd-272wd\" (UID: \"63f70bd9-cc95-455b-98ab-7d313d37f1a8\") " pod="kuadrant-system/authorino-6f6944bcdd-272wd" Oct 30 18:07:56.734338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.734242 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhx8\" (UniqueName: \"kubernetes.io/projected/63f70bd9-cc95-455b-98ab-7d313d37f1a8-kube-api-access-wwhx8\") pod \"authorino-6f6944bcdd-272wd\" (UID: \"63f70bd9-cc95-455b-98ab-7d313d37f1a8\") " pod="kuadrant-system/authorino-6f6944bcdd-272wd" Oct 30 18:07:56.759794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.759708 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhx8\" (UniqueName: \"kubernetes.io/projected/63f70bd9-cc95-455b-98ab-7d313d37f1a8-kube-api-access-wwhx8\") pod \"authorino-6f6944bcdd-272wd\" (UID: \"63f70bd9-cc95-455b-98ab-7d313d37f1a8\") " pod="kuadrant-system/authorino-6f6944bcdd-272wd" Oct 30 18:07:56.877873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.877793 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-272wd" Oct 30 18:07:56.893638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.893559 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-272wd"] Oct 30 18:07:56.923175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.923097 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fgwbw"] Oct 30 18:07:56.929528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.929474 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:56.936619 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.936523 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf512f40-ddb0-4c5d-8dbf-9f0581043504-tls-cert\") pod \"authorino-6889cbd8c4-fgwbw\" (UID: \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\") " pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:56.937097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.937059 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpch\" (UniqueName: \"kubernetes.io/projected/bf512f40-ddb0-4c5d-8dbf-9f0581043504-kube-api-access-dkpch\") pod \"authorino-6889cbd8c4-fgwbw\" (UID: \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\") " pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:56.945837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.945778 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 18:07:56.949086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.949024 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fgwbw"] Oct 30 18:07:56.981374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:56.981291 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fgwbw"] Oct 30 18:07:56.981920 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:07:56.981859 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dkpch tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" podUID="bf512f40-ddb0-4c5d-8dbf-9f0581043504" Oct 30 18:07:57.030338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.030243 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dbc5c4f76-2skb9"] Oct 30 18:07:57.039642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.039552 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpch\" (UniqueName: \"kubernetes.io/projected/bf512f40-ddb0-4c5d-8dbf-9f0581043504-kube-api-access-dkpch\") pod \"authorino-6889cbd8c4-fgwbw\" (UID: \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\") " pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:57.039984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.039682 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf512f40-ddb0-4c5d-8dbf-9f0581043504-tls-cert\") pod \"authorino-6889cbd8c4-fgwbw\" (UID: \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\") " pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:57.046558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.046493 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dbc5c4f76-2skb9"] Oct 30 18:07:57.047245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.047202 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf512f40-ddb0-4c5d-8dbf-9f0581043504-tls-cert\") pod \"authorino-6889cbd8c4-fgwbw\" (UID: \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\") " pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:57.047808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.047260 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:07:57.066206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.066138 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpch\" (UniqueName: \"kubernetes.io/projected/bf512f40-ddb0-4c5d-8dbf-9f0581043504-kube-api-access-dkpch\") pod \"authorino-6889cbd8c4-fgwbw\" (UID: \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\") " pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:57.140833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.140603 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xck\" (UniqueName: \"kubernetes.io/projected/8c47a709-c81b-4365-9f37-5408803b2d82-kube-api-access-d4xck\") pod \"authorino-7dbc5c4f76-2skb9\" (UID: \"8c47a709-c81b-4365-9f37-5408803b2d82\") " pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:07:57.140833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.140820 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c47a709-c81b-4365-9f37-5408803b2d82-tls-cert\") pod \"authorino-7dbc5c4f76-2skb9\" (UID: \"8c47a709-c81b-4365-9f37-5408803b2d82\") " pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:07:57.227570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.227274 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-272wd"] Oct 30 18:07:57.232263 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:07:57.232192 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63f70bd9_cc95_455b_98ab_7d313d37f1a8.slice/crio-978f744b5fff1c1477f271da0be62851859e2eeb774cd4d95a4b074ad30d6e43 WatchSource:0}: Error finding container 978f744b5fff1c1477f271da0be62851859e2eeb774cd4d95a4b074ad30d6e43: Status 404 returned error can't find the container with id 978f744b5fff1c1477f271da0be62851859e2eeb774cd4d95a4b074ad30d6e43 Oct 30 18:07:57.242606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.242531 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xck\" (UniqueName: \"kubernetes.io/projected/8c47a709-c81b-4365-9f37-5408803b2d82-kube-api-access-d4xck\") pod \"authorino-7dbc5c4f76-2skb9\" (UID: \"8c47a709-c81b-4365-9f37-5408803b2d82\") " pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:07:57.243011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.242694 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c47a709-c81b-4365-9f37-5408803b2d82-tls-cert\") pod \"authorino-7dbc5c4f76-2skb9\" (UID: \"8c47a709-c81b-4365-9f37-5408803b2d82\") " pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:07:57.247645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.247561 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c47a709-c81b-4365-9f37-5408803b2d82-tls-cert\") pod \"authorino-7dbc5c4f76-2skb9\" (UID: \"8c47a709-c81b-4365-9f37-5408803b2d82\") " pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:07:57.266033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.265964 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xck\" (UniqueName: \"kubernetes.io/projected/8c47a709-c81b-4365-9f37-5408803b2d82-kube-api-access-d4xck\") pod \"authorino-7dbc5c4f76-2skb9\" (UID: \"8c47a709-c81b-4365-9f37-5408803b2d82\") " pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:07:57.397215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.397059 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:07:57.736336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.736249 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dbc5c4f76-2skb9"] Oct 30 18:07:57.738852 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:07:57.738794 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c47a709_c81b_4365_9f37_5408803b2d82.slice/crio-9e75f9303d833c5b61b658f28566b232ff708004ceaf495f3c02e955cfc30ca2 WatchSource:0}: Error finding container 9e75f9303d833c5b61b658f28566b232ff708004ceaf495f3c02e955cfc30ca2: Status 404 returned error can't find the container with id 9e75f9303d833c5b61b658f28566b232ff708004ceaf495f3c02e955cfc30ca2 Oct 30 18:07:57.828916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.828834 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-272wd" event={"ID":"63f70bd9-cc95-455b-98ab-7d313d37f1a8","Type":"ContainerStarted","Data":"91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804"} Oct 30 18:07:57.829344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.828994 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-272wd" event={"ID":"63f70bd9-cc95-455b-98ab-7d313d37f1a8","Type":"ContainerStarted","Data":"978f744b5fff1c1477f271da0be62851859e2eeb774cd4d95a4b074ad30d6e43"} Oct 30 18:07:57.829344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.829024 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-272wd" podUID="63f70bd9-cc95-455b-98ab-7d313d37f1a8" containerName="authorino" containerID="cri-o://91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804" gracePeriod=30 Oct 30 18:07:57.833386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.833301 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" event={"ID":"8c47a709-c81b-4365-9f37-5408803b2d82","Type":"ContainerStarted","Data":"9e75f9303d833c5b61b658f28566b232ff708004ceaf495f3c02e955cfc30ca2"} Oct 30 18:07:57.833768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.833539 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:57.873876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.873752 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-272wd" podStartSLOduration=1.429544744 podStartE2EDuration="1.873719251s" totalImagesPullingTime="444.174507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:57.872318541 +0000 UTC m=+372.260846659" watchObservedRunningTime="2026-10-30 18:07:57.873719251 +0000 UTC m=+372.262247663" Oct 30 18:07:57.941088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.940957 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:57.951090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.951033 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf512f40-ddb0-4c5d-8dbf-9f0581043504-kube-api-access-dkpch\" (UniqueName: \"kubernetes.io/projected/bf512f40-ddb0-4c5d-8dbf-9f0581043504-kube-api-access-dkpch\") pod \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\" (UID: \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\") " Oct 30 18:07:57.951401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.951217 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf512f40-ddb0-4c5d-8dbf-9f0581043504-tls-cert\" (UniqueName: \"kubernetes.io/secret/bf512f40-ddb0-4c5d-8dbf-9f0581043504-tls-cert\") pod \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\" (UID: \"bf512f40-ddb0-4c5d-8dbf-9f0581043504\") " Oct 30 18:07:57.954676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.954589 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf512f40-ddb0-4c5d-8dbf-9f0581043504-tls-cert" pod "bf512f40-ddb0-4c5d-8dbf-9f0581043504" (UID: "bf512f40-ddb0-4c5d-8dbf-9f0581043504"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:57.954676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:57.954615 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf512f40-ddb0-4c5d-8dbf-9f0581043504-kube-api-access-dkpch" pod "bf512f40-ddb0-4c5d-8dbf-9f0581043504" (UID: "bf512f40-ddb0-4c5d-8dbf-9f0581043504"). InnerVolumeSpecName "kube-api-access-dkpch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:58.053425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.053314 5624 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf512f40-ddb0-4c5d-8dbf-9f0581043504-tls-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:07:58.053425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.053385 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkpch\" (UniqueName: \"kubernetes.io/projected/bf512f40-ddb0-4c5d-8dbf-9f0581043504-kube-api-access-dkpch\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:07:58.109683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.109612 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7"] Oct 30 18:07:58.115588 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.115525 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.157925 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.158149 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbr6m\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-kube-api-access-jbr6m\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.158309 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.158385 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285dc7ba-c00a-408b-a32b-8f87279f2b52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.158428 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.158464 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.158529 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.158579 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.158622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.158608 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.159507 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.159450 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7"] Oct 30 18:07:58.188580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.188520 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-272wd" Oct 30 18:07:58.259614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.259535 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f70bd9-cc95-455b-98ab-7d313d37f1a8-kube-api-access-wwhx8\" (UniqueName: \"kubernetes.io/projected/63f70bd9-cc95-455b-98ab-7d313d37f1a8-kube-api-access-wwhx8\") pod \"63f70bd9-cc95-455b-98ab-7d313d37f1a8\" (UID: \"63f70bd9-cc95-455b-98ab-7d313d37f1a8\") " Oct 30 18:07:58.260032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.259848 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.259990 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbr6m\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-kube-api-access-jbr6m\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260047 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260122 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285dc7ba-c00a-408b-a32b-8f87279f2b52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260293 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260213 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260293 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260240 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260302 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260352 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260382 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.260854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260768 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.261062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.260851 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.261183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.261132 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.261183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.261165 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.261880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.261803 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285dc7ba-c00a-408b-a32b-8f87279f2b52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.264830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.264770 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.266952 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.266877 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.267873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.267814 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f70bd9-cc95-455b-98ab-7d313d37f1a8-kube-api-access-wwhx8" pod "63f70bd9-cc95-455b-98ab-7d313d37f1a8" (UID: "63f70bd9-cc95-455b-98ab-7d313d37f1a8"). InnerVolumeSpecName "kube-api-access-wwhx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:58.283539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.283463 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbr6m\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-kube-api-access-jbr6m\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.283539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.283517 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-vhrw7\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.362064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.361983 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhx8\" (UniqueName: \"kubernetes.io/projected/63f70bd9-cc95-455b-98ab-7d313d37f1a8-kube-api-access-wwhx8\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:07:58.458582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.458499 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:07:58.830346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.829002 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7"] Oct 30 18:07:58.832448 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:07:58.832386 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285dc7ba_c00a_408b_a32b_8f87279f2b52.slice/crio-9fec8ee56ff5b82759b599cc43d18212b529a9a05d769769b5e26afd8175f5f0 WatchSource:0}: Error finding container 9fec8ee56ff5b82759b599cc43d18212b529a9a05d769769b5e26afd8175f5f0: Status 404 returned error can't find the container with id 9fec8ee56ff5b82759b599cc43d18212b529a9a05d769769b5e26afd8175f5f0 Oct 30 18:07:58.837708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.837624 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:07:58.838116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.837807 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:07:58.838116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.837870 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:07:58.856210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.856137 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" event={"ID":"8c47a709-c81b-4365-9f37-5408803b2d82","Type":"ContainerStarted","Data":"8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a"} Oct 30 18:07:58.860753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.860677 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" event={"ID":"285dc7ba-c00a-408b-a32b-8f87279f2b52","Type":"ContainerStarted","Data":"9fec8ee56ff5b82759b599cc43d18212b529a9a05d769769b5e26afd8175f5f0"} Oct 30 18:07:58.864614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.864539 5624 generic.go:356] "Generic (PLEG): container finished" podID="63f70bd9-cc95-455b-98ab-7d313d37f1a8" containerID="91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804" exitCode=2 Oct 30 18:07:58.865017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.864882 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fgwbw" Oct 30 18:07:58.875161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.875045 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-272wd" event={"ID":"63f70bd9-cc95-455b-98ab-7d313d37f1a8","Type":"ContainerDied","Data":"91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804"} Oct 30 18:07:58.875161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.875132 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-272wd" event={"ID":"63f70bd9-cc95-455b-98ab-7d313d37f1a8","Type":"ContainerDied","Data":"978f744b5fff1c1477f271da0be62851859e2eeb774cd4d95a4b074ad30d6e43"} Oct 30 18:07:58.875161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.875163 5624 scope.go:122] "RemoveContainer" containerID="91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804" Oct 30 18:07:58.877666 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.877601 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-272wd" Oct 30 18:07:58.900667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.900544 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" podStartSLOduration=2.412527853 podStartE2EDuration="2.900504006s" totalImagesPullingTime="487.976153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:58.899010686 +0000 UTC m=+373.287538804" watchObservedRunningTime="2026-10-30 18:07:58.900504006 +0000 UTC m=+373.289032124" Oct 30 18:07:58.938721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.938660 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mwclg"] Oct 30 18:07:58.969385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.969323 5624 scope.go:122] "RemoveContainer" containerID="91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804" Oct 30 18:07:58.978191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.974403 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" podUID="d2563a61-e976-47ab-8d03-1772aa98150f" containerName="authorino" containerID="cri-o://38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784" gracePeriod=30 Oct 30 18:07:58.979204 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:07:58.979130 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804\": container with ID starting with 91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804 not found: ID does not exist" containerID="91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804" Oct 30 18:07:58.982203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.981159 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804"} err="failed to get container status \"91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804\": rpc error: code = NotFound desc = could not find container \"91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804\": container with ID starting with 91447f1c34cab24da198f86f8c950a5e3e96fdbd70040d2674dfc9fccff7f804 not found: ID does not exist" Oct 30 18:07:58.986090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.985324 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-272wd"] Oct 30 18:07:58.994197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:58.994134 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-272wd"] Oct 30 18:07:59.066725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.066528 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fgwbw"] Oct 30 18:07:59.073546 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.072653 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fgwbw"] Oct 30 18:07:59.259423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.259343 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:07:59.259862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.259471 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:07:59.376885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.376821 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" Oct 30 18:07:59.386426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.385759 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2563a61-e976-47ab-8d03-1772aa98150f-kube-api-access-8hfmz\" (UniqueName: \"kubernetes.io/projected/d2563a61-e976-47ab-8d03-1772aa98150f-kube-api-access-8hfmz\") pod \"d2563a61-e976-47ab-8d03-1772aa98150f\" (UID: \"d2563a61-e976-47ab-8d03-1772aa98150f\") " Oct 30 18:07:59.396089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.395988 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2563a61-e976-47ab-8d03-1772aa98150f-kube-api-access-8hfmz" pod "d2563a61-e976-47ab-8d03-1772aa98150f" (UID: "d2563a61-e976-47ab-8d03-1772aa98150f"). InnerVolumeSpecName "kube-api-access-8hfmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:59.488437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.488335 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hfmz\" (UniqueName: \"kubernetes.io/projected/d2563a61-e976-47ab-8d03-1772aa98150f-kube-api-access-8hfmz\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:07:59.822569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.820702 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f70bd9-cc95-455b-98ab-7d313d37f1a8" path="/var/lib/kubelet/pods/63f70bd9-cc95-455b-98ab-7d313d37f1a8/volumes" Oct 30 18:07:59.822569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.821845 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf512f40-ddb0-4c5d-8dbf-9f0581043504" path="/var/lib/kubelet/pods/bf512f40-ddb0-4c5d-8dbf-9f0581043504/volumes" Oct 30 18:07:59.878064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.877836 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" event={"ID":"285dc7ba-c00a-408b-a32b-8f87279f2b52","Type":"ContainerStarted","Data":"1820103603e1a4ecdc728e715ea386966c5cd80495306d26dd8980ab2dd2eff9"} Oct 30 18:07:59.894983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.884052 5624 generic.go:356] "Generic (PLEG): container finished" podID="d2563a61-e976-47ab-8d03-1772aa98150f" containerID="38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784" exitCode=0 Oct 30 18:07:59.894983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.884152 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" event={"ID":"d2563a61-e976-47ab-8d03-1772aa98150f","Type":"ContainerDied","Data":"38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784"} Oct 30 18:07:59.894983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.884258 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" event={"ID":"d2563a61-e976-47ab-8d03-1772aa98150f","Type":"ContainerDied","Data":"80b1e8747a7b37414ddd3bc69cc76fa8a908bac8cf260e124111585d1356dfc5"} Oct 30 18:07:59.894983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.884296 5624 scope.go:122] "RemoveContainer" containerID="38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784" Oct 30 18:07:59.894983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.887011 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mwclg" Oct 30 18:07:59.928388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.928334 5624 scope.go:122] "RemoveContainer" containerID="38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784" Oct 30 18:07:59.929351 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:07:59.929300 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784\": container with ID starting with 38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784 not found: ID does not exist" containerID="38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784" Oct 30 18:07:59.929607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:07:59.929368 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784"} err="failed to get container status \"38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784\": rpc error: code = NotFound desc = could not find container \"38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784\": container with ID starting with 38b0bc6df36f464660b1b9cbd5c8123ab04cb421329563576e8d1766cb948784 not found: ID does not exist" Oct 30 18:08:00.014714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.014545 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podStartSLOduration=2.014503865 podStartE2EDuration="2.014503865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:59.952786562 +0000 UTC m=+374.341314679" watchObservedRunningTime="2026-10-30 18:08:00.014503865 +0000 UTC m=+374.403031986" Oct 30 18:08:00.016100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.016026 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mwclg"] Oct 30 18:08:00.019252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.019194 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mwclg"] Oct 30 18:08:00.459173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.459069 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:08:00.462716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.462641 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:08:00.463113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.462777 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 18:08:00.922210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.922150 5624 generic.go:356] "Generic (PLEG): container finished" podID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerID="747691b8f4a81b18fc3f0ae5750bc015f12b7f0c30115abef0eacdd26ab86854" exitCode=0 Oct 30 18:08:00.922997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.922243 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" event={"ID":"28da4ab2-eb1e-458b-93a5-c063d30768de","Type":"ContainerDied","Data":"747691b8f4a81b18fc3f0ae5750bc015f12b7f0c30115abef0eacdd26ab86854"} Oct 30 18:08:00.923091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.923033 5624 scope.go:122] "RemoveContainer" containerID="747691b8f4a81b18fc3f0ae5750bc015f12b7f0c30115abef0eacdd26ab86854" Oct 30 18:08:00.925365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.925300 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:08:00.925575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.925449 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:08:00.925575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:00.925497 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:08:01.460095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:01.460033 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:08:01.460515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:01.460120 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 18:08:01.838024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:01.837733 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2563a61-e976-47ab-8d03-1772aa98150f" path="/var/lib/kubelet/pods/d2563a61-e976-47ab-8d03-1772aa98150f/volumes" Oct 30 18:08:01.944603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:01.944525 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" event={"ID":"28da4ab2-eb1e-458b-93a5-c063d30768de","Type":"ContainerStarted","Data":"7c77088cf56846869f48ac89fbd3f53282dcff24be84fd255232deff10832404"} Oct 30 18:08:02.459370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:02.459288 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:08:02.459717 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:02.459392 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 18:08:02.704630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:02.704555 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:08:02.705406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:02.705353 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 18:08:02.705775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:02.705723 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 18:08:03.460067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:03.460009 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:08:03.460750 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:03.460104 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 18:08:03.706889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:03.706794 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 18:08:03.707528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:03.707468 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 18:08:04.459411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:04.459342 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:08:04.459758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:04.459438 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 18:08:04.701066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:04.700987 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:04.714516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:04.714286 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 18:08:04.715186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:04.715079 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 18:08:04.804533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:04.804407 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:05.459477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:05.459399 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:08:05.459818 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:05.459499 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 18:08:05.707988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:05.707871 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 18:08:05.708969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:05.708042 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 18:08:06.459915 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:06.459830 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.460294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:06.459995 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 18:08:06.644911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:06.644841 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh"] Oct 30 18:08:06.645336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:06.645286 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" podUID="614dc6f6-4f2d-4e15-8219-f67e449f9739" containerName="maas-api" containerID="cri-o://71dd491b1b89cb0631e0892155f75828c3d3ccc0e8a146fe6eedfda05a4560ee" gracePeriod=30 Oct 30 18:08:06.715417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:06.715233 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.715417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:06.715369 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 18:08:07.015406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.015350 5624 generic.go:356] "Generic (PLEG): container finished" podID="614dc6f6-4f2d-4e15-8219-f67e449f9739" containerID="71dd491b1b89cb0631e0892155f75828c3d3ccc0e8a146fe6eedfda05a4560ee" exitCode=0 Oct 30 18:08:07.015542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.015432 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" event={"ID":"614dc6f6-4f2d-4e15-8219-f67e449f9739","Type":"ContainerDied","Data":"71dd491b1b89cb0631e0892155f75828c3d3ccc0e8a146fe6eedfda05a4560ee"} Oct 30 18:08:07.047027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.046908 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:08:07.156040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.155979 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614dc6f6-4f2d-4e15-8219-f67e449f9739-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614dc6f6-4f2d-4e15-8219-f67e449f9739-openshift-service-ca\") pod \"614dc6f6-4f2d-4e15-8219-f67e449f9739\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " Oct 30 18:08:07.156040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.156040 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614dc6f6-4f2d-4e15-8219-f67e449f9739-kube-api-access-glq6m\" (UniqueName: \"kubernetes.io/projected/614dc6f6-4f2d-4e15-8219-f67e449f9739-kube-api-access-glq6m\") pod \"614dc6f6-4f2d-4e15-8219-f67e449f9739\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " Oct 30 18:08:07.156483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.156109 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-metrics-certs\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-metrics-certs\") pod \"614dc6f6-4f2d-4e15-8219-f67e449f9739\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " Oct 30 18:08:07.156483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.156153 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-maas-api-tls\") pod \"614dc6f6-4f2d-4e15-8219-f67e449f9739\" (UID: \"614dc6f6-4f2d-4e15-8219-f67e449f9739\") " Oct 30 18:08:07.156608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.156551 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614dc6f6-4f2d-4e15-8219-f67e449f9739-openshift-service-ca" pod "614dc6f6-4f2d-4e15-8219-f67e449f9739" (UID: "614dc6f6-4f2d-4e15-8219-f67e449f9739"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:08:07.159952 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.159853 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-maas-api-tls" pod "614dc6f6-4f2d-4e15-8219-f67e449f9739" (UID: "614dc6f6-4f2d-4e15-8219-f67e449f9739"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:07.159952 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.159871 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-metrics-certs" pod "614dc6f6-4f2d-4e15-8219-f67e449f9739" (UID: "614dc6f6-4f2d-4e15-8219-f67e449f9739"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:08:07.160269 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.159874 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614dc6f6-4f2d-4e15-8219-f67e449f9739-kube-api-access-glq6m" pod "614dc6f6-4f2d-4e15-8219-f67e449f9739" (UID: "614dc6f6-4f2d-4e15-8219-f67e449f9739"). InnerVolumeSpecName "kube-api-access-glq6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:07.257557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.257471 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:08:07.257557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.257538 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/614dc6f6-4f2d-4e15-8219-f67e449f9739-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:08:07.257557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.257557 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glq6m\" (UniqueName: \"kubernetes.io/projected/614dc6f6-4f2d-4e15-8219-f67e449f9739-kube-api-access-glq6m\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:08:07.257557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.257572 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/614dc6f6-4f2d-4e15-8219-f67e449f9739-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:08:07.468306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.468240 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:08:07.468923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.468896 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:08:07.470476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.470438 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:08:07.709783 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.709716 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 18:08:07.710179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:07.709811 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 18:08:08.027649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:08.027580 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" event={"ID":"614dc6f6-4f2d-4e15-8219-f67e449f9739","Type":"ContainerDied","Data":"851f5e33e55cb201b8ae93e89b7d4a9993be10dc97187a7e9246c2f76b713651"} Oct 30 18:08:08.027649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:08.027618 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh" Oct 30 18:08:08.028600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:08.027723 5624 scope.go:122] "RemoveContainer" containerID="71dd491b1b89cb0631e0892155f75828c3d3ccc0e8a146fe6eedfda05a4560ee" Oct 30 18:08:08.110610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:08.110368 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh"] Oct 30 18:08:08.114398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:08.114288 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-2bgxh"] Oct 30 18:08:08.717677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:08.717617 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:08:08.718347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:08.718312 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:08:08.720033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:08.720004 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:08:09.792473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:09.792381 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614dc6f6-4f2d-4e15-8219-f67e449f9739" path="/var/lib/kubelet/pods/614dc6f6-4f2d-4e15-8219-f67e449f9739/volumes" Oct 30 18:08:19.930503 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:08:19.930423 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:08:28.197859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.197782 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2"] Oct 30 18:08:28.198610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198578 5624 cpu_manager.go:414] "containerMap: removing container" podUID="63f70bd9-cc95-455b-98ab-7d313d37f1a8" containerName="authorino" Oct 30 18:08:28.198672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198614 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f70bd9-cc95-455b-98ab-7d313d37f1a8" containerName="authorino" Oct 30 18:08:28.198672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198650 5624 cpu_manager.go:414] "containerMap: removing container" podUID="614dc6f6-4f2d-4e15-8219-f67e449f9739" containerName="maas-api" Oct 30 18:08:28.198672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198656 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614dc6f6-4f2d-4e15-8219-f67e449f9739" containerName="maas-api" Oct 30 18:08:28.199340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198680 5624 cpu_manager.go:414] "containerMap: removing container" podUID="d2563a61-e976-47ab-8d03-1772aa98150f" containerName="authorino" Oct 30 18:08:28.199340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198688 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2563a61-e976-47ab-8d03-1772aa98150f" containerName="authorino" Oct 30 18:08:28.199340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198783 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="614dc6f6-4f2d-4e15-8219-f67e449f9739" containerName="maas-api" Oct 30 18:08:28.199340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198800 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2563a61-e976-47ab-8d03-1772aa98150f" containerName="authorino" Oct 30 18:08:28.199340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.198809 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f70bd9-cc95-455b-98ab-7d313d37f1a8" containerName="authorino" Oct 30 18:08:28.203246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.203192 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.226644 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.226576 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2"] Oct 30 18:08:28.342768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.342685 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.342768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.342778 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.343203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.343043 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvsm5\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-kube-api-access-hvsm5\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.343203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.343106 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.343203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.343153 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.343307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.343219 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.343307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.343258 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c84ac3-01a9-4a7a-b659-31d591531f66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.343391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.343372 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.343436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.343418 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445276 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445377 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445444 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvsm5\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-kube-api-access-hvsm5\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445478 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445517 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445547 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445577 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c84ac3-01a9-4a7a-b659-31d591531f66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445641 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.445820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.445677 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.446325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.446215 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.446325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.446235 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.446518 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.446481 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.446707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.446676 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.446755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.446712 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c84ac3-01a9-4a7a-b659-31d591531f66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.449593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.449466 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.449812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.449702 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.470325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.470255 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.470683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.470599 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvsm5\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-kube-api-access-hvsm5\") pod \"maas-default-gateway-openshift-default-77f556d647-bb5z2\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.533018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.532954 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:28.888165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.888082 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2"] Oct 30 18:08:28.894473 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:08:28.894407 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c84ac3_01a9_4a7a_b659_31d591531f66.slice/crio-fc9987bc69e9e7e183d083c2fc6d4bf02ff8daad3974b61d97b5338aa594e237 WatchSource:0}: Error finding container fc9987bc69e9e7e183d083c2fc6d4bf02ff8daad3974b61d97b5338aa594e237: Status 404 returned error can't find the container with id fc9987bc69e9e7e183d083c2fc6d4bf02ff8daad3974b61d97b5338aa594e237 Oct 30 18:08:28.898388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.898321 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:08:28.898647 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.898463 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:08:28.898647 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:28.898502 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:08:29.217509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:29.217146 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" event={"ID":"21c84ac3-01a9-4a7a-b659-31d591531f66","Type":"ContainerStarted","Data":"e3f1aa7c53f2eedf115342ac2fc45e5b1a4938a6e17ff3c6a57225ec02f706a2"} Oct 30 18:08:29.217509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:29.217282 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" event={"ID":"21c84ac3-01a9-4a7a-b659-31d591531f66","Type":"ContainerStarted","Data":"fc9987bc69e9e7e183d083c2fc6d4bf02ff8daad3974b61d97b5338aa594e237"} Oct 30 18:08:29.265740 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:29.265588 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podStartSLOduration=1.2655437809999999 podStartE2EDuration="1.265543781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:29.263898145 +0000 UTC m=+403.652426263" watchObservedRunningTime="2026-10-30 18:08:29.265543781 +0000 UTC m=+403.654071899" Oct 30 18:08:29.533442 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:29.533259 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:30.537048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:30.536996 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:08:30.537918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:30.537089 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:08:31.534241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:31.534159 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:08:31.534602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:31.534261 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:08:32.533920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:32.533834 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:08:32.534689 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:32.533989 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:08:33.533877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:33.533815 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:08:33.534294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:33.533923 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:08:34.534146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:34.534066 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:08:34.534502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:34.534182 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:08:34.610796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:34.610712 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:34.721026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:34.720969 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:35.534244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:35.534177 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:08:35.534632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:35.534284 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 18:08:36.542003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:36.541877 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:36.542668 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:36.542593 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:36.544881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:36.544844 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:08:39.318842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:39.318777 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:47.243409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:47.243329 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:47.332028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:47.331923 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:49.931822 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:08:49.931751 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:08:53.524995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.524879 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7wj9m"] Oct 30 18:08:53.533310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.533211 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.550302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.550238 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7wj9m"] Oct 30 18:08:53.559368 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.559305 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8g42t\"" Oct 30 18:08:53.559368 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.559330 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:08:53.559769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.559551 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:08:53.567100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.567040 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 18:08:53.646660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.646553 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:53.694327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.694241 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-home\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.694667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.694471 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.694667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.694543 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmh8g\" (UniqueName: \"kubernetes.io/projected/d2333eaa-171b-4dd4-a978-24ba23defed7-kube-api-access-nmh8g\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.694667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.694605 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2333eaa-171b-4dd4-a978-24ba23defed7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.694667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.694642 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.694820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.694689 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.796374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.796192 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-home\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.796374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.796302 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.796374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.796359 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmh8g\" (UniqueName: \"kubernetes.io/projected/d2333eaa-171b-4dd4-a978-24ba23defed7-kube-api-access-nmh8g\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.796858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.796399 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2333eaa-171b-4dd4-a978-24ba23defed7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.796858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.796433 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.796858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.796462 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.797055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.796912 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-home\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.797055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.796929 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.797055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.797028 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.800466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.800406 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2333eaa-171b-4dd4-a978-24ba23defed7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.800868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.800844 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2333eaa-171b-4dd4-a978-24ba23defed7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.821498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.821425 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmh8g\" (UniqueName: \"kubernetes.io/projected/d2333eaa-171b-4dd4-a978-24ba23defed7-kube-api-access-nmh8g\") pod \"echo-simulator-kserve-64b6f8f96c-7wj9m\" (UID: \"d2333eaa-171b-4dd4-a978-24ba23defed7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:53.865187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:53.865113 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:08:54.217926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:54.217861 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-7wj9m"] Oct 30 18:08:54.220850 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:08:54.220778 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2333eaa_171b_4dd4_a978_24ba23defed7.slice/crio-843e94820416e0907f3a395a7fcb4b644aadaec1ecd9b440bbf7d9cf82458db8 WatchSource:0}: Error finding container 843e94820416e0907f3a395a7fcb4b644aadaec1ecd9b440bbf7d9cf82458db8: Status 404 returned error can't find the container with id 843e94820416e0907f3a395a7fcb4b644aadaec1ecd9b440bbf7d9cf82458db8 Oct 30 18:08:54.423060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:54.423008 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:08:54.424572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:54.424491 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" event={"ID":"d2333eaa-171b-4dd4-a978-24ba23defed7","Type":"ContainerStarted","Data":"843e94820416e0907f3a395a7fcb4b644aadaec1ecd9b440bbf7d9cf82458db8"} Oct 30 18:08:55.389560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.389483 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk"] Oct 30 18:08:55.428551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.419457 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk"] Oct 30 18:08:55.429121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.421677 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.437096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.436889 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:55.501974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.501853 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 18:08:55.604175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.603439 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a32196-f4a2-41c3-b028-25e792821d05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.604175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.603716 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwsrp\" (UniqueName: \"kubernetes.io/projected/32a32196-f4a2-41c3-b028-25e792821d05-kube-api-access-dwsrp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.604175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.603824 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.604175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.603874 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.604175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.604003 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.604175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.604047 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.706009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.705521 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwsrp\" (UniqueName: \"kubernetes.io/projected/32a32196-f4a2-41c3-b028-25e792821d05-kube-api-access-dwsrp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.706009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.705661 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.706009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.705709 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.706009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.705785 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.706009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.705836 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.707477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.706842 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.707477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.706843 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.707477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.707115 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.707477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.707150 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a32196-f4a2-41c3-b028-25e792821d05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.711859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.711789 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a32196-f4a2-41c3-b028-25e792821d05-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.715241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.715108 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a32196-f4a2-41c3-b028-25e792821d05-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.738602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.738527 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwsrp\" (UniqueName: \"kubernetes.io/projected/32a32196-f4a2-41c3-b028-25e792821d05-kube-api-access-dwsrp\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk\" (UID: \"32a32196-f4a2-41c3-b028-25e792821d05\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:55.883916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:55.883833 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:08:56.266177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:56.266120 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk"] Oct 30 18:08:56.279250 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:08:56.279152 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a32196_f4a2_41c3_b028_25e792821d05.slice/crio-42ed9f33728a76bf74645bb7e67a91bc7589cd89c8c469addd8eca2ef5ff21ba WatchSource:0}: Error finding container 42ed9f33728a76bf74645bb7e67a91bc7589cd89c8c469addd8eca2ef5ff21ba: Status 404 returned error can't find the container with id 42ed9f33728a76bf74645bb7e67a91bc7589cd89c8c469addd8eca2ef5ff21ba Oct 30 18:08:56.460502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:56.460419 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" event={"ID":"32a32196-f4a2-41c3-b028-25e792821d05","Type":"ContainerStarted","Data":"42ed9f33728a76bf74645bb7e67a91bc7589cd89c8c469addd8eca2ef5ff21ba"} Oct 30 18:08:56.848159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:56.848072 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd"] Oct 30 18:08:56.893223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:56.893142 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:56.917166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:56.917072 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd"] Oct 30 18:08:56.917543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:56.917312 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:56.945061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:56.944376 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 18:08:57.086991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.082302 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.086991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.082495 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.086991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.082676 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da2b19fd-2648-4e80-bde1-16211807bc16-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.086991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.082768 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.086991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.082874 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9zdh\" (UniqueName: \"kubernetes.io/projected/da2b19fd-2648-4e80-bde1-16211807bc16-kube-api-access-h9zdh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.086991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.083031 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.188991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.184899 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da2b19fd-2648-4e80-bde1-16211807bc16-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.209382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.209307 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.216006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.215827 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.231096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.209920 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zdh\" (UniqueName: \"kubernetes.io/projected/da2b19fd-2648-4e80-bde1-16211807bc16-kube-api-access-h9zdh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.231876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.231802 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.232365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.232326 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.232622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.232595 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.235068 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.233534 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.262984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.237074 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.263595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.254141 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da2b19fd-2648-4e80-bde1-16211807bc16-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.278250 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.278162 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da2b19fd-2648-4e80-bde1-16211807bc16-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.281966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.281060 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zdh\" (UniqueName: \"kubernetes.io/projected/da2b19fd-2648-4e80-bde1-16211807bc16-kube-api-access-h9zdh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd\" (UID: \"da2b19fd-2648-4e80-bde1-16211807bc16\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.281966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.281423 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:08:57.876174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:57.876031 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd"] Oct 30 18:08:58.454052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.452317 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:08:58.525600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.525444 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" event={"ID":"da2b19fd-2648-4e80-bde1-16211807bc16","Type":"ContainerStarted","Data":"459cf3a15d1ea7fc48f157c47cabbfc350c931ef66c095587c61a7b5a0382fd7"} Oct 30 18:08:58.670139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.669863 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz"] Oct 30 18:08:58.677826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.677756 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.696849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.696762 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz"] Oct 30 18:08:58.699562 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.698580 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-png8m\" (UniqueName: \"kubernetes.io/projected/79e88ba7-0418-4268-b1b3-e50946e634eb-kube-api-access-png8m\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.700362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.700313 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79e88ba7-0418-4268-b1b3-e50946e634eb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.700858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.700738 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.701205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.701178 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.701449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.701424 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.701926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.701883 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.705310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.705128 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 18:08:58.745017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.744914 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:58.804651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.804577 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.804992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.804824 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-png8m\" (UniqueName: \"kubernetes.io/projected/79e88ba7-0418-4268-b1b3-e50946e634eb-kube-api-access-png8m\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.804992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.804969 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79e88ba7-0418-4268-b1b3-e50946e634eb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.805116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.805003 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.805116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.805043 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.805116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.805090 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.805835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.805795 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.806277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.806231 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.808723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.808663 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.813806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.813719 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79e88ba7-0418-4268-b1b3-e50946e634eb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.832225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.832090 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79e88ba7-0418-4268-b1b3-e50946e634eb-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:58.832667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:58.832619 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-png8m\" (UniqueName: \"kubernetes.io/projected/79e88ba7-0418-4268-b1b3-e50946e634eb-kube-api-access-png8m\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz\" (UID: \"79e88ba7-0418-4268-b1b3-e50946e634eb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:59.041103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:59.040387 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:08:59.529928 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:08:59.529835 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz"] Oct 30 18:08:59.541815 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:08:59.541712 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e88ba7_0418_4268_b1b3_e50946e634eb.slice/crio-c125bfa8005ca1fbfb050c04801cd94ac4f3f4ed6b0b852d6a14bac83889dd36 WatchSource:0}: Error finding container c125bfa8005ca1fbfb050c04801cd94ac4f3f4ed6b0b852d6a14bac83889dd36: Status 404 returned error can't find the container with id c125bfa8005ca1fbfb050c04801cd94ac4f3f4ed6b0b852d6a14bac83889dd36 Oct 30 18:09:00.282994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.278825 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw"] Oct 30 18:09:00.377207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.336810 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw"] Oct 30 18:09:00.377207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.337078 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.377207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.348367 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.377207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.348468 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.377207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.348531 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e069f9a5-f97d-45a9-a64a-d47a4118f251-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.377207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.348570 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nl7\" (UniqueName: \"kubernetes.io/projected/e069f9a5-f97d-45a9-a64a-d47a4118f251-kube-api-access-r7nl7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.377207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.348709 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.377207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.348825 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.425993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.407062 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.427267 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.461562 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.461656 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.461727 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e069f9a5-f97d-45a9-a64a-d47a4118f251-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.461767 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nl7\" (UniqueName: \"kubernetes.io/projected/e069f9a5-f97d-45a9-a64a-d47a4118f251-kube-api-access-r7nl7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.462004 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.462143 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.463294 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.471859 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.570149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.473040 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.873363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.873270 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nl7\" (UniqueName: \"kubernetes.io/projected/e069f9a5-f97d-45a9-a64a-d47a4118f251-kube-api-access-r7nl7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.877148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.876924 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e069f9a5-f97d-45a9-a64a-d47a4118f251-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.898073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.891525 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e069f9a5-f97d-45a9-a64a-d47a4118f251-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7gjfw\" (UID: \"e069f9a5-f97d-45a9-a64a-d47a4118f251\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.915048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.910538 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:00.933441 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:00.929412 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" event={"ID":"79e88ba7-0418-4268-b1b3-e50946e634eb","Type":"ContainerStarted","Data":"c125bfa8005ca1fbfb050c04801cd94ac4f3f4ed6b0b852d6a14bac83889dd36"} Oct 30 18:09:01.431013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:01.430459 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw"] Oct 30 18:09:01.459359 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:09:01.459255 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode069f9a5_f97d_45a9_a64a_d47a4118f251.slice/crio-c221ac95f388c07a6cbfc0c5e7a63d9b516c0bc133d46d687b9996a6a7032324 WatchSource:0}: Error finding container c221ac95f388c07a6cbfc0c5e7a63d9b516c0bc133d46d687b9996a6a7032324: Status 404 returned error can't find the container with id c221ac95f388c07a6cbfc0c5e7a63d9b516c0bc133d46d687b9996a6a7032324 Oct 30 18:09:01.977299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:01.977207 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" event={"ID":"e069f9a5-f97d-45a9-a64a-d47a4118f251","Type":"ContainerStarted","Data":"c221ac95f388c07a6cbfc0c5e7a63d9b516c0bc133d46d687b9996a6a7032324"} Oct 30 18:09:01.987017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:01.984824 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" event={"ID":"32a32196-f4a2-41c3-b028-25e792821d05","Type":"ContainerStarted","Data":"96683167e25351b40d9a1f3a5131f6e580002fac06e258170ca638e03189e9df"} Oct 30 18:09:01.998879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:01.997371 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" event={"ID":"d2333eaa-171b-4dd4-a978-24ba23defed7","Type":"ContainerStarted","Data":"5068dbb6d784a96f195749a53cc6ecbfa01b48a6a8f934e6aca64ce918886ca8"} Oct 30 18:09:02.036719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:02.036232 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podStartSLOduration=1.536657677 podStartE2EDuration="7.036202586s" totalImagesPullingTime="5.499544909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:02.035020791 +0000 UTC m=+436.423548906" watchObservedRunningTime="2026-10-30 18:09:02.036202586 +0000 UTC m=+436.424730708" Oct 30 18:09:02.095624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:02.091737 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podStartSLOduration=3.033342601 podStartE2EDuration="9.091689939s" totalImagesPullingTime="6.058347338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:02.087453532 +0000 UTC m=+436.475981648" watchObservedRunningTime="2026-10-30 18:09:02.091689939 +0000 UTC m=+436.480218059" Oct 30 18:09:02.606582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:02.605605 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:02.871402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:02.871103 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:09:02.884644 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:02.884420 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:09:03.032978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:03.030170 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" event={"ID":"79e88ba7-0418-4268-b1b3-e50946e634eb","Type":"ContainerStarted","Data":"7049812b443f3d3433f6f6e01263b66fc4b041c45096e05e79cba2dd42bfa97e"} Oct 30 18:09:03.043047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:03.042283 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:09:03.098475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:03.088894 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podStartSLOduration=2.320493763 podStartE2EDuration="5.088850716s" totalImagesPullingTime="2.768356953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:03.076883409 +0000 UTC m=+437.465411564" watchObservedRunningTime="2026-10-30 18:09:03.088850716 +0000 UTC m=+437.477378835" Oct 30 18:09:03.131999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:03.129873 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podStartSLOduration=2.223568324 podStartE2EDuration="7.129838728s" totalImagesPullingTime="4.906270404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:03.128379727 +0000 UTC m=+437.516907846" watchObservedRunningTime="2026-10-30 18:09:03.129838728 +0000 UTC m=+437.518366845" Oct 30 18:09:03.288247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:03.288154 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:09:03.867581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:03.867484 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:09:03.884006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:03.883875 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:09:04.044576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.044488 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk"] Oct 30 18:09:04.084627 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.084545 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk"] Oct 30 18:09:04.085554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.085506 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.162266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.146730 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 18:09:04.169687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.164000 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.169687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.164145 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.169687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.164235 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xwn\" (UniqueName: \"kubernetes.io/projected/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-kube-api-access-95xwn\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.169687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.164411 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.169687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.164456 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.169687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.164488 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.169687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.169468 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:04.279833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.279743 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xwn\" (UniqueName: \"kubernetes.io/projected/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-kube-api-access-95xwn\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.280764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.280501 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.281331 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.281286 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.281587 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.281567 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.282044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.282008 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.316014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.282310 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.316014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.282368 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.316014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.283168 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.316014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.312359 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.403011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.332429 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" event={"ID":"da2b19fd-2648-4e80-bde1-16211807bc16","Type":"ContainerStarted","Data":"b771750b90f1dadb0e453c990c876d9a66b7e38f018ab9b5822015d128cebe0e"} Oct 30 18:09:04.434249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.417160 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" event={"ID":"e069f9a5-f97d-45a9-a64a-d47a4118f251","Type":"ContainerStarted","Data":"5dfad3441102011203a840d05f5c43bf19c545f44a76530a6f5c89c4587b366c"} Oct 30 18:09:04.480998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.475850 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.480998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.479192 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:09:04.480998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.480429 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.536370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.536182 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xwn\" (UniqueName: \"kubernetes.io/projected/eba05bdf-704e-4dc8-9da9-3e52b6f0a285-kube-api-access-95xwn\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-lz5rk\" (UID: \"eba05bdf-704e-4dc8-9da9-3e52b6f0a285\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.613988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.593266 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podStartSLOduration=2.804961669 podStartE2EDuration="4.593220253s" totalImagesPullingTime="1.788258584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:04.520703813 +0000 UTC m=+438.909231932" watchObservedRunningTime="2026-10-30 18:09:04.593220253 +0000 UTC m=+438.981748372" Oct 30 18:09:04.777043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.775554 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:04.914309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:04.914217 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:05.542612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:05.516006 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:05.713717 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:05.713582 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk"] Oct 30 18:09:05.720553 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:09:05.720479 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba05bdf_704e_4dc8_9da9_3e52b6f0a285.slice/crio-6b6a7c8d178a44443c5dab6111d3c5047d92ed545de899912c8326367539cd69 WatchSource:0}: Error finding container 6b6a7c8d178a44443c5dab6111d3c5047d92ed545de899912c8326367539cd69: Status 404 returned error can't find the container with id 6b6a7c8d178a44443c5dab6111d3c5047d92ed545de899912c8326367539cd69 Oct 30 18:09:05.884655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:05.884583 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:09:05.901864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:05.901794 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:09:06.492493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:06.492410 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" event={"ID":"eba05bdf-704e-4dc8-9da9-3e52b6f0a285","Type":"ContainerStarted","Data":"d265ffbba3ebe2a46628357b54239959c3ddd60ef57eb2470497bb7b12698440"} Oct 30 18:09:06.493281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:06.492536 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" event={"ID":"eba05bdf-704e-4dc8-9da9-3e52b6f0a285","Type":"ContainerStarted","Data":"6b6a7c8d178a44443c5dab6111d3c5047d92ed545de899912c8326367539cd69"} Oct 30 18:09:06.510603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:06.510532 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:09:06.542026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:06.541848 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podStartSLOduration=2.322900782 podStartE2EDuration="2.541811135s" totalImagesPullingTime="218.910353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:06.538525357 +0000 UTC m=+440.927053474" watchObservedRunningTime="2026-10-30 18:09:06.541811135 +0000 UTC m=+440.930339253" Oct 30 18:09:06.776589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:06.776426 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:07.282748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:07.282663 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:09:07.305758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:07.305686 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:09:07.521697 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:07.521556 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:09:07.571048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:07.570867 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:09:09.043239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:09.043155 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:09:09.066859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:09.066787 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:09:09.637921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:09.637855 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:09:10.916971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:10.916848 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:10.959310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:10.959243 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:11.681921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:11.681851 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:09:13.334593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.334512 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm"] Oct 30 18:09:13.341167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.341104 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.369790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.369711 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm"] Oct 30 18:09:13.473824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.473749 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.473824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.473838 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.474315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.474021 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.474315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.474200 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.474315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.474245 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.474511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.474359 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fsj\" (UniqueName: \"kubernetes.io/projected/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-kube-api-access-98fsj\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.474549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.474503 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.474659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.474633 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.474788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.474774 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.576987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.576875 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.577398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.577032 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.577398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.577084 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.577398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.577141 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.577398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.577250 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.577398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.577283 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.577398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.577326 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fsj\" (UniqueName: \"kubernetes.io/projected/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-kube-api-access-98fsj\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.577398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.577365 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.577726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.577453 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.578568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.578470 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.578568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.578521 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.578920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.578626 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.579092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.579046 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.579451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.579413 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.581889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.581825 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.582530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.582477 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.603519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.603343 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fsj\" (UniqueName: \"kubernetes.io/projected/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-kube-api-access-98fsj\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.604164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.604121 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c9434e-7f80-41bf-92c8-c8e8dbb4671b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4bknm\" (UID: \"50c9434e-7f80-41bf-92c8-c8e8dbb4671b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:13.679995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:13.679858 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:14.045733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.045659 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm"] Oct 30 18:09:14.048130 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:09:14.048058 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c9434e_7f80_41bf_92c8_c8e8dbb4671b.slice/crio-1c404d9f65b0e20f018a8edc0481ba9cc876a80e8b52a7111b98634d18acd7cd WatchSource:0}: Error finding container 1c404d9f65b0e20f018a8edc0481ba9cc876a80e8b52a7111b98634d18acd7cd: Status 404 returned error can't find the container with id 1c404d9f65b0e20f018a8edc0481ba9cc876a80e8b52a7111b98634d18acd7cd Oct 30 18:09:14.052433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.052335 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:09:14.052727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.052522 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:09:14.052727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.052579 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:09:14.683364 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.683281 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" event={"ID":"50c9434e-7f80-41bf-92c8-c8e8dbb4671b","Type":"ContainerStarted","Data":"3313e71bfec9fb5ce3ea6422020fa2ed5965368c068bcc2b974ea28f8a99211b"} Oct 30 18:09:14.684168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.683402 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" event={"ID":"50c9434e-7f80-41bf-92c8-c8e8dbb4671b","Type":"ContainerStarted","Data":"1c404d9f65b0e20f018a8edc0481ba9cc876a80e8b52a7111b98634d18acd7cd"} Oct 30 18:09:14.761612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.761478 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podStartSLOduration=1.761439278 podStartE2EDuration="1.761439278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:14.744848635 +0000 UTC m=+449.133376753" watchObservedRunningTime="2026-10-30 18:09:14.761439278 +0000 UTC m=+449.149967638" Oct 30 18:09:14.776210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.776134 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:14.802266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:14.802196 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:15.680839 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:15.680760 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:15.692994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:15.689254 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:09:15.692994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:15.689422 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 18:09:15.722414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:15.722343 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:09:16.681770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:16.681690 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:09:16.682252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:16.681823 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 18:09:17.698988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:17.698579 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:09:17.698988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:17.698712 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 18:09:18.680704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:18.680598 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:09:18.681157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:18.680750 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 18:09:19.681546 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:19.681469 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:09:19.682427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:19.681646 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 18:09:19.954469 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:09:19.954399 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:09:20.680744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:20.680674 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:09:20.681137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:20.680804 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 18:09:21.681598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:21.681510 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:09:21.682734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:21.681633 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 18:09:22.681300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:22.681218 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:09:22.681708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:22.681333 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 18:09:24.679384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:24.679210 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:24.680084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:24.679992 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:24.681731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:24.681697 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" Oct 30 18:09:25.774928 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:25.774852 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cb9789cff-xpn9q"] Oct 30 18:09:25.780220 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:25.780165 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cb9789cff-xpn9q" Oct 30 18:09:25.797626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:25.797563 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cb9789cff-xpn9q"] Oct 30 18:09:25.922264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:25.922150 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8fx8\" (UniqueName: \"kubernetes.io/projected/81799468-0dff-48d9-ba4c-dc4d0c29cf40-kube-api-access-l8fx8\") pod \"authorino-cb9789cff-xpn9q\" (UID: \"81799468-0dff-48d9-ba4c-dc4d0c29cf40\") " pod="kuadrant-system/authorino-cb9789cff-xpn9q" Oct 30 18:09:25.922662 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:25.922412 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81799468-0dff-48d9-ba4c-dc4d0c29cf40-tls-cert\") pod \"authorino-cb9789cff-xpn9q\" (UID: \"81799468-0dff-48d9-ba4c-dc4d0c29cf40\") " pod="kuadrant-system/authorino-cb9789cff-xpn9q" Oct 30 18:09:26.024812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.024708 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8fx8\" (UniqueName: \"kubernetes.io/projected/81799468-0dff-48d9-ba4c-dc4d0c29cf40-kube-api-access-l8fx8\") pod \"authorino-cb9789cff-xpn9q\" (UID: \"81799468-0dff-48d9-ba4c-dc4d0c29cf40\") " pod="kuadrant-system/authorino-cb9789cff-xpn9q" Oct 30 18:09:26.024812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.024823 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81799468-0dff-48d9-ba4c-dc4d0c29cf40-tls-cert\") pod \"authorino-cb9789cff-xpn9q\" (UID: \"81799468-0dff-48d9-ba4c-dc4d0c29cf40\") " pod="kuadrant-system/authorino-cb9789cff-xpn9q" Oct 30 18:09:26.029239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.029092 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/81799468-0dff-48d9-ba4c-dc4d0c29cf40-tls-cert\") pod \"authorino-cb9789cff-xpn9q\" (UID: \"81799468-0dff-48d9-ba4c-dc4d0c29cf40\") " pod="kuadrant-system/authorino-cb9789cff-xpn9q" Oct 30 18:09:26.056537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.056450 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8fx8\" (UniqueName: \"kubernetes.io/projected/81799468-0dff-48d9-ba4c-dc4d0c29cf40-kube-api-access-l8fx8\") pod \"authorino-cb9789cff-xpn9q\" (UID: \"81799468-0dff-48d9-ba4c-dc4d0c29cf40\") " pod="kuadrant-system/authorino-cb9789cff-xpn9q" Oct 30 18:09:26.106473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.106376 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cb9789cff-xpn9q" Oct 30 18:09:26.293618 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.293441 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:26.455085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.455023 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cb9789cff-xpn9q"] Oct 30 18:09:26.457731 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:09:26.457646 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81799468_0dff_48d9_ba4c_dc4d0c29cf40.slice/crio-0653d330e308610c9638a355fa51731cb1b99f1a86973d3e996b975670293d71 WatchSource:0}: Error finding container 0653d330e308610c9638a355fa51731cb1b99f1a86973d3e996b975670293d71: Status 404 returned error can't find the container with id 0653d330e308610c9638a355fa51731cb1b99f1a86973d3e996b975670293d71 Oct 30 18:09:26.460532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.460469 5624 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:09:26.861576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:26.861491 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cb9789cff-xpn9q" event={"ID":"81799468-0dff-48d9-ba4c-dc4d0c29cf40","Type":"ContainerStarted","Data":"0653d330e308610c9638a355fa51731cb1b99f1a86973d3e996b975670293d71"} Oct 30 18:09:27.109015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:27.108914 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:27.957669 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:27.957578 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cb9789cff-xpn9q" event={"ID":"81799468-0dff-48d9-ba4c-dc4d0c29cf40","Type":"ContainerStarted","Data":"2d0a60f338d99057de1880cf012d28e5f1358c026545a9b1b486cd020267483c"} Oct 30 18:09:28.032155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.032030 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-cb9789cff-xpn9q" podStartSLOduration=2.615302402 podStartE2EDuration="3.031998775s" totalImagesPullingTime="416.696373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:28.024680555 +0000 UTC m=+462.413208673" watchObservedRunningTime="2026-10-30 18:09:28.031998775 +0000 UTC m=+462.420526892" Oct 30 18:09:28.125280 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.125113 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dbc5c4f76-2skb9"] Oct 30 18:09:28.125663 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.125553 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" podUID="8c47a709-c81b-4365-9f37-5408803b2d82" containerName="authorino" containerID="cri-o://8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a" gracePeriod=30 Oct 30 18:09:28.497781 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.497704 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:09:28.623319 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.623248 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c47a709-c81b-4365-9f37-5408803b2d82-tls-cert\" (UniqueName: \"kubernetes.io/secret/8c47a709-c81b-4365-9f37-5408803b2d82-tls-cert\") pod \"8c47a709-c81b-4365-9f37-5408803b2d82\" (UID: \"8c47a709-c81b-4365-9f37-5408803b2d82\") " Oct 30 18:09:28.623714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.623379 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c47a709-c81b-4365-9f37-5408803b2d82-kube-api-access-d4xck\" (UniqueName: \"kubernetes.io/projected/8c47a709-c81b-4365-9f37-5408803b2d82-kube-api-access-d4xck\") pod \"8c47a709-c81b-4365-9f37-5408803b2d82\" (UID: \"8c47a709-c81b-4365-9f37-5408803b2d82\") " Oct 30 18:09:28.627471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.627378 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c47a709-c81b-4365-9f37-5408803b2d82-kube-api-access-d4xck" pod "8c47a709-c81b-4365-9f37-5408803b2d82" (UID: "8c47a709-c81b-4365-9f37-5408803b2d82"). InnerVolumeSpecName "kube-api-access-d4xck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:28.644655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.644535 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c47a709-c81b-4365-9f37-5408803b2d82-tls-cert" pod "8c47a709-c81b-4365-9f37-5408803b2d82" (UID: "8c47a709-c81b-4365-9f37-5408803b2d82"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:28.726410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.726334 5624 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8c47a709-c81b-4365-9f37-5408803b2d82-tls-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:09:28.726410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.726408 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xck\" (UniqueName: \"kubernetes.io/projected/8c47a709-c81b-4365-9f37-5408803b2d82-kube-api-access-d4xck\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:09:28.969862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.969664 5624 generic.go:356] "Generic (PLEG): container finished" podID="8c47a709-c81b-4365-9f37-5408803b2d82" containerID="8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a" exitCode=0 Oct 30 18:09:28.969862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.969754 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" Oct 30 18:09:28.969862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.969760 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" event={"ID":"8c47a709-c81b-4365-9f37-5408803b2d82","Type":"ContainerDied","Data":"8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a"} Oct 30 18:09:28.969862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.969844 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dbc5c4f76-2skb9" event={"ID":"8c47a709-c81b-4365-9f37-5408803b2d82","Type":"ContainerDied","Data":"9e75f9303d833c5b61b658f28566b232ff708004ceaf495f3c02e955cfc30ca2"} Oct 30 18:09:28.970874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.969879 5624 scope.go:122] "RemoveContainer" containerID="8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a" Oct 30 18:09:28.991199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.990442 5624 scope.go:122] "RemoveContainer" containerID="8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a" Oct 30 18:09:28.991528 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:09:28.991432 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a\": container with ID starting with 8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a not found: ID does not exist" containerID="8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a" Oct 30 18:09:28.991613 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:28.991520 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a"} err="failed to get container status \"8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a\": rpc error: code = NotFound desc = could not find container \"8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a\": container with ID starting with 8d08ae6821d444301ae95f54cb864542b588f0067df24e7cab955f90992b479a not found: ID does not exist" Oct 30 18:09:29.034448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:29.034372 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dbc5c4f76-2skb9"] Oct 30 18:09:29.046286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:29.046230 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7dbc5c4f76-2skb9"] Oct 30 18:09:29.802583 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:29.802514 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c47a709-c81b-4365-9f37-5408803b2d82" path="/var/lib/kubelet/pods/8c47a709-c81b-4365-9f37-5408803b2d82/volumes" Oct 30 18:09:30.857395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:30.857328 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:30.963714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:30.963636 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:31.212679 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.212598 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-j6n87"] Oct 30 18:09:31.219537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.219458 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8c47a709-c81b-4365-9f37-5408803b2d82" containerName="authorino" Oct 30 18:09:31.220196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.220151 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c47a709-c81b-4365-9f37-5408803b2d82" containerName="authorino" Oct 30 18:09:31.220720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.220678 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c47a709-c81b-4365-9f37-5408803b2d82" containerName="authorino" Oct 30 18:09:31.233050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.232852 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:31.261508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.261442 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 18:09:31.281876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.280855 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tsv\" (UniqueName: \"kubernetes.io/projected/d13122e5-4950-40fc-8bb5-046b9fa77766-kube-api-access-n2tsv\") pod \"payload-processing-5f78687d5b-j6n87\" (UID: \"d13122e5-4950-40fc-8bb5-046b9fa77766\") " pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:31.281876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.281074 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13122e5-4950-40fc-8bb5-046b9fa77766-plugins-config-volume\") pod \"payload-processing-5f78687d5b-j6n87\" (UID: \"d13122e5-4950-40fc-8bb5-046b9fa77766\") " pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:31.284081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.283549 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jbvcp\"" Oct 30 18:09:31.342971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.342258 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-j6n87"] Oct 30 18:09:31.382409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.382338 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tsv\" (UniqueName: \"kubernetes.io/projected/d13122e5-4950-40fc-8bb5-046b9fa77766-kube-api-access-n2tsv\") pod \"payload-processing-5f78687d5b-j6n87\" (UID: \"d13122e5-4950-40fc-8bb5-046b9fa77766\") " pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:31.382741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.382440 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13122e5-4950-40fc-8bb5-046b9fa77766-plugins-config-volume\") pod \"payload-processing-5f78687d5b-j6n87\" (UID: \"d13122e5-4950-40fc-8bb5-046b9fa77766\") " pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:31.383625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.383576 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13122e5-4950-40fc-8bb5-046b9fa77766-plugins-config-volume\") pod \"payload-processing-5f78687d5b-j6n87\" (UID: \"d13122e5-4950-40fc-8bb5-046b9fa77766\") " pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:31.408725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.408080 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tsv\" (UniqueName: \"kubernetes.io/projected/d13122e5-4950-40fc-8bb5-046b9fa77766-kube-api-access-n2tsv\") pod \"payload-processing-5f78687d5b-j6n87\" (UID: \"d13122e5-4950-40fc-8bb5-046b9fa77766\") " pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:31.627121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:31.627034 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:32.025549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:32.025472 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-j6n87"] Oct 30 18:09:32.042356 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:09:32.033244 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13122e5_4950_40fc_8bb5_046b9fa77766.slice/crio-d011d3d3399470423809ad11e03f7f2f3e89315d4c6700439ab77399dac7e1f6 WatchSource:0}: Error finding container d011d3d3399470423809ad11e03f7f2f3e89315d4c6700439ab77399dac7e1f6: Status 404 returned error can't find the container with id d011d3d3399470423809ad11e03f7f2f3e89315d4c6700439ab77399dac7e1f6 Oct 30 18:09:33.053170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:33.053000 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" event={"ID":"d13122e5-4950-40fc-8bb5-046b9fa77766","Type":"ContainerStarted","Data":"d011d3d3399470423809ad11e03f7f2f3e89315d4c6700439ab77399dac7e1f6"} Oct 30 18:09:36.147911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:36.147820 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" event={"ID":"d13122e5-4950-40fc-8bb5-046b9fa77766","Type":"ContainerStarted","Data":"5c166b2c502c9f0b9308c751f137ac0ecd23dd3dab1a9d5fa8121951f2ac6222"} Oct 30 18:09:36.148851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:36.148027 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:36.196770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:36.196633 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" podStartSLOduration=1.63519869 podStartE2EDuration="5.196592582s" totalImagesPullingTime="3.561393892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:36.195446071 +0000 UTC m=+470.583974190" watchObservedRunningTime="2026-10-30 18:09:36.196592582 +0000 UTC m=+470.585120700" Oct 30 18:09:39.457008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:39.456905 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:40.902851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:40.902762 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm"] Oct 30 18:09:40.908514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:40.908461 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:40.927543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:40.927478 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm"] Oct 30 18:09:40.934558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:40.934504 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 18:09:40.937737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:40.937686 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mdm7d\"" Oct 30 18:09:40.937737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:40.937701 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 18:09:41.018371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:41.018288 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4326dbaf-621c-4d40-9f86-e29dbdd1b649-metrics-certs\") pod \"ai-gateway-controller-5bcd7466d5-kzdwm\" (UID: \"4326dbaf-621c-4d40-9f86-e29dbdd1b649\") " pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:41.018698 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:41.018605 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjxlp\" (UniqueName: \"kubernetes.io/projected/4326dbaf-621c-4d40-9f86-e29dbdd1b649-kube-api-access-bjxlp\") pod \"ai-gateway-controller-5bcd7466d5-kzdwm\" (UID: \"4326dbaf-621c-4d40-9f86-e29dbdd1b649\") " pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:41.120142 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:41.120067 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjxlp\" (UniqueName: \"kubernetes.io/projected/4326dbaf-621c-4d40-9f86-e29dbdd1b649-kube-api-access-bjxlp\") pod \"ai-gateway-controller-5bcd7466d5-kzdwm\" (UID: \"4326dbaf-621c-4d40-9f86-e29dbdd1b649\") " pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:41.120614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:41.120213 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4326dbaf-621c-4d40-9f86-e29dbdd1b649-metrics-certs\") pod \"ai-gateway-controller-5bcd7466d5-kzdwm\" (UID: \"4326dbaf-621c-4d40-9f86-e29dbdd1b649\") " pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:41.124725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:41.124649 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4326dbaf-621c-4d40-9f86-e29dbdd1b649-metrics-certs\") pod \"ai-gateway-controller-5bcd7466d5-kzdwm\" (UID: \"4326dbaf-621c-4d40-9f86-e29dbdd1b649\") " pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:41.144026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:41.143926 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjxlp\" (UniqueName: \"kubernetes.io/projected/4326dbaf-621c-4d40-9f86-e29dbdd1b649-kube-api-access-bjxlp\") pod \"ai-gateway-controller-5bcd7466d5-kzdwm\" (UID: \"4326dbaf-621c-4d40-9f86-e29dbdd1b649\") " pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:41.235432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:41.235271 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:41.578642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:41.578580 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm"] Oct 30 18:09:41.581084 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:09:41.581022 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4326dbaf_621c_4d40_9f86_e29dbdd1b649.slice/crio-0bebc1b2e5c30c08e236e4fb6e9f5c62477dc57748079e229811f2005da4410b WatchSource:0}: Error finding container 0bebc1b2e5c30c08e236e4fb6e9f5c62477dc57748079e229811f2005da4410b: Status 404 returned error can't find the container with id 0bebc1b2e5c30c08e236e4fb6e9f5c62477dc57748079e229811f2005da4410b Oct 30 18:09:42.205056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:42.205000 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" event={"ID":"4326dbaf-621c-4d40-9f86-e29dbdd1b649","Type":"ContainerStarted","Data":"0bebc1b2e5c30c08e236e4fb6e9f5c62477dc57748079e229811f2005da4410b"} Oct 30 18:09:43.218349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:43.218281 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" event={"ID":"4326dbaf-621c-4d40-9f86-e29dbdd1b649","Type":"ContainerStarted","Data":"ac887b7f81d5246cf180f0dcbcc8a0e1abf5a087cede64d6e96c2b99b17ac6a3"} Oct 30 18:09:43.219101 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:43.218391 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:43.260714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:43.260553 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" podStartSLOduration=1.800855047 podStartE2EDuration="3.260510121s" totalImagesPullingTime="1.459655074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:43.258920034 +0000 UTC m=+477.647448150" watchObservedRunningTime="2026-10-30 18:09:43.260510121 +0000 UTC m=+477.649038239" Oct 30 18:09:47.163996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:47.163905 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" Oct 30 18:09:49.932428 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:09:49.932344 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:09:54.231252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:54.231164 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:09:55.972447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:55.972365 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:56.485685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:56.485611 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:58.158345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:58.158266 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:58.295907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:58.295831 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:59.770179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:59.770109 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:59.878170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:09:59.878104 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:01.119211 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:01.119125 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:01.219977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:01.219871 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:19.932433 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:10:19.932360 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:10:22.466725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:22.466642 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:22.658378 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:22.658309 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:22.946913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:22.946808 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw"] Oct 30 18:10:22.960039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:22.959918 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:22.975605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:22.975503 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw"] Oct 30 18:10:22.981559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:22.981474 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b6cf1-openshift-default-dockercfg-l4bh4\"" Oct 30 18:10:23.009496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.009328 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:23.066462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.066367 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.066871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.066481 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.066871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.066530 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.066871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.066780 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a9e445-860a-46c5-80bd-6ed297d09b77-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.066871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.066864 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.067188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.066951 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.067326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.067280 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfctc\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-kube-api-access-hfctc\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.067448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.067421 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.067527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.067510 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.122133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.122056 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:23.169228 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169121 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfctc\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-kube-api-access-hfctc\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.169228 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169257 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.169228 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169309 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.169828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169347 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.169828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169407 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.169828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169430 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.169828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169518 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a9e445-860a-46c5-80bd-6ed297d09b77-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.169828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169604 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.169828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.169663 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.170314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.170142 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.170314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.170194 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.170314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.170192 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.170626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.170559 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.171083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.171035 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a9e445-860a-46c5-80bd-6ed297d09b77-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.173903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.173837 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.174307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.174271 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.202190 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.202044 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.202526 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.202473 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfctc\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-kube-api-access-hfctc\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.247929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.247850 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:23.294269 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.294190 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:23.311488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.311403 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w"] Oct 30 18:10:23.324023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.322271 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.350364 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.350291 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w"] Oct 30 18:10:23.363277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.362670 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:23.374266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.374090 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68n5b\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-kube-api-access-68n5b\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.375336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.374614 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.375336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.374760 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac309921-c2c5-4c6a-8635-79dea80cfadf-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.375336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.374848 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.375336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.374888 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.375336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.375011 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.375336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.375055 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.375336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.375110 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.375336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.375146 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.390156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.390088 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2fa0-openshift-default-dockercfg-vw5sp\"" Oct 30 18:10:23.464582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.464396 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx"] Oct 30 18:10:23.477610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.473883 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.478888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.478830 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.478913 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.479100 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68n5b\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-kube-api-access-68n5b\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.479159 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.479261 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac309921-c2c5-4c6a-8635-79dea80cfadf-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.479374 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.479420 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.479575 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.479664 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.479851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.479792 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.480252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.480210 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.480397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.480353 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.481477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.481423 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac309921-c2c5-4c6a-8635-79dea80cfadf-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.483485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.483379 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.490160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.488611 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.490160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.490005 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx"] Oct 30 18:10:23.491581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.491487 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:23.492323 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.492277 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fdf4da-openshift-default-dockercfg-gnh4d\"" Oct 30 18:10:23.492893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.492755 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.525235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.525161 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.525820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.525712 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68n5b\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-kube-api-access-68n5b\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.583409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.583211 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.583409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.583323 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.583409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.583363 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.583409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.583425 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zx9q\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-kube-api-access-8zx9q\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.584007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.583721 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.584007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.583877 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4d072e-56c8-4f49-966c-18445c91eea2-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.584309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.584264 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.584597 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.584565 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.584723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.584642 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.617496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.617423 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:23.686600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.686503 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4d072e-56c8-4f49-966c-18445c91eea2-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.686600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.686599 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.687187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.686765 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.687187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.686798 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.687187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.686848 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.687187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.686893 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.687187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.686922 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.687187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.686980 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zx9q\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-kube-api-access-8zx9q\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.687187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.687138 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.688816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.688216 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.688816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.688316 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4d072e-56c8-4f49-966c-18445c91eea2-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.688816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.688362 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.688816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.688708 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.689928 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.689870 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.691917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.691861 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.692311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.692276 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.700032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.699489 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:23.706793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.706722 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw"] Oct 30 18:10:23.708811 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:10:23.708745 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a9e445_860a_46c5_80bd_6ed297d09b77.slice/crio-cc360df749b357d3024640b7b8a8c7eb840ad9563468f82eec9c63de5058ce3f WatchSource:0}: Error finding container cc360df749b357d3024640b7b8a8c7eb840ad9563468f82eec9c63de5058ce3f: Status 404 returned error can't find the container with id cc360df749b357d3024640b7b8a8c7eb840ad9563468f82eec9c63de5058ce3f Oct 30 18:10:23.718390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.718156 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.722582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.722438 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2"] Oct 30 18:10:23.725659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.725597 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zx9q\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-kube-api-access-8zx9q\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.730730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.730663 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.737157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.737043 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:23.738239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.738155 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:23.742361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.742253 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:23.764789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.751218 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ef6ac-openshift-default-dockercfg-vxmbz\"" Oct 30 18:10:23.764789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.755352 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2"] Oct 30 18:10:23.771297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.771224 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:23.788915 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.788847 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.789566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.789524 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.789834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.789813 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.790145 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.790118 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59z8w\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-kube-api-access-59z8w\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.790481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.790379 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5293e2-5b63-430f-aa36-9733a6a35dd3-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.790481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.790465 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.790591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.790528 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.790636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.790602 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.790723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.790690 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.817719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.817641 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:23.884873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.880798 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:23.896812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.896261 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.896812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.896408 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.896812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.896604 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.896812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.896642 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.896812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.896713 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.896812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.896801 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59z8w\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-kube-api-access-59z8w\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.897502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.896881 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5293e2-5b63-430f-aa36-9733a6a35dd3-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.897502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.896989 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.897502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.897061 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.897502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.897157 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.898038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.897990 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.898418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.898387 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.898685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.898660 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.913653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.910875 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5293e2-5b63-430f-aa36-9733a6a35dd3-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.914259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.914154 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.915522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.915366 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.923706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.923630 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.924641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.924513 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59z8w\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-kube-api-access-59z8w\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:23.990774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:23.990577 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:24.149536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.149260 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:24.153888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.153112 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w"] Oct 30 18:10:24.203467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.203385 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.203667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.203581 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.203667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.203659 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.233119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.233057 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx"] Oct 30 18:10:24.244473 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:10:24.241430 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4d072e_56c8_4f49_966c_18445c91eea2.slice/crio-60396a22644890f48f5370ac2efc8f8636847a61fda30ee234b879dba9c5a080 WatchSource:0}: Error finding container 60396a22644890f48f5370ac2efc8f8636847a61fda30ee234b879dba9c5a080: Status 404 returned error can't find the container with id 60396a22644890f48f5370ac2efc8f8636847a61fda30ee234b879dba9c5a080 Oct 30 18:10:24.262063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.261979 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.262435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.262123 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.262435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.262184 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.556076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.556031 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2"] Oct 30 18:10:24.566522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.566428 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.567086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.567052 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.567299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.567281 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:10:24.608790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.608710 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" event={"ID":"cc4d072e-56c8-4f49-966c-18445c91eea2","Type":"ContainerStarted","Data":"82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae"} Oct 30 18:10:24.609177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.608832 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" event={"ID":"cc4d072e-56c8-4f49-966c-18445c91eea2","Type":"ContainerStarted","Data":"60396a22644890f48f5370ac2efc8f8636847a61fda30ee234b879dba9c5a080"} Oct 30 18:10:24.616042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.615961 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" event={"ID":"56a9e445-860a-46c5-80bd-6ed297d09b77","Type":"ContainerStarted","Data":"54f17b7248f7563e7f291f5874d4a5b435a9dcfe1dc08af6b6723bc42f9c38b9"} Oct 30 18:10:24.616523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.616082 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" event={"ID":"56a9e445-860a-46c5-80bd-6ed297d09b77","Type":"ContainerStarted","Data":"cc360df749b357d3024640b7b8a8c7eb840ad9563468f82eec9c63de5058ce3f"} Oct 30 18:10:24.621263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.621185 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" event={"ID":"df5293e2-5b63-430f-aa36-9733a6a35dd3","Type":"ContainerStarted","Data":"e325dd9539631d835e7b2cc42247c7620e517eba08667395f7f330e1e833f59d"} Oct 30 18:10:24.625232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.624830 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" event={"ID":"ac309921-c2c5-4c6a-8635-79dea80cfadf","Type":"ContainerStarted","Data":"e6c0619e00ae7c0869b255b050920b47562d7bf2e5f0ca002d42bdebb0acd96e"} Oct 30 18:10:24.625232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.625050 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" event={"ID":"ac309921-c2c5-4c6a-8635-79dea80cfadf","Type":"ContainerStarted","Data":"e05ddd6d8017905a68859a2b61fa1f381e39ac80492b109fe4a31b538c98af37"} Oct 30 18:10:24.665850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.664434 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podStartSLOduration=1.6644019399999999 podStartE2EDuration="1.66440194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:24.659822729 +0000 UTC m=+519.048350834" watchObservedRunningTime="2026-10-30 18:10:24.66440194 +0000 UTC m=+519.052930057" Oct 30 18:10:24.700386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.700295 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:24.719490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.713633 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podStartSLOduration=1.713595669 podStartE2EDuration="1.713595669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:24.711190433 +0000 UTC m=+519.099718550" watchObservedRunningTime="2026-10-30 18:10:24.713595669 +0000 UTC m=+519.102123787" Oct 30 18:10:24.762343 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.762181 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podStartSLOduration=2.762133611 podStartE2EDuration="2.762133611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:24.760885445 +0000 UTC m=+519.149413657" watchObservedRunningTime="2026-10-30 18:10:24.762133611 +0000 UTC m=+519.150661732" Oct 30 18:10:24.818601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:24.818399 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:25.295207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:25.295135 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:25.307510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:25.307443 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:25.639427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:25.639344 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" event={"ID":"df5293e2-5b63-430f-aa36-9733a6a35dd3","Type":"ContainerStarted","Data":"0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52"} Oct 30 18:10:25.640423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:25.640385 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:25.641880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:25.641833 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:10:25.694462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:25.694290 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podStartSLOduration=2.694251626 podStartE2EDuration="2.694251626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:25.68975649 +0000 UTC m=+520.078284608" watchObservedRunningTime="2026-10-30 18:10:25.694251626 +0000 UTC m=+520.082779739" Oct 30 18:10:25.711911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:25.711840 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:25.829805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:25.829676 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:26.151097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.151034 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:26.163120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.163061 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:26.453553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.453394 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:26.562287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.562208 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:26.650443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.650147 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:26.651251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.650763 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:26.651251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.650809 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:26.651855 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.651813 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:10:26.652085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.651977 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:10:26.652233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:26.652203 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:10:27.186608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:27.186552 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:27.304871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:27.304787 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:27.445454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:27.445294 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:27.572912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:27.572843 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:27.703904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:27.703701 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:27.845365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:27.845303 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:27.962767 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:27.962578 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:28.088261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:28.088176 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:28.214328 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:28.214148 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:28.918929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:28.918864 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:29.059393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:29.059328 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:30.915693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:30.915611 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.029268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.029198 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.442625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.442540 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9"] Oct 30 18:10:31.448571 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.448518 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.477790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.477654 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-fde2c5-serving-cert\"" Oct 30 18:10:31.478245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.477921 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:10:31.478245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.478154 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:10:31.478245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.478151 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:10:31.478483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.478249 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:10:31.484427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.484340 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9"] Oct 30 18:10:31.530614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.530541 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.541320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.541241 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6bnj\" (UniqueName: \"kubernetes.io/projected/8fef5526-096c-45de-a569-4e80570d9b28-kube-api-access-r6bnj\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.541712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.541428 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.541712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.541638 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fef5526-096c-45de-a569-4e80570d9b28-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.541831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.541806 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-metrics-certs\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.626685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.626603 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.643803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.643726 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6bnj\" (UniqueName: \"kubernetes.io/projected/8fef5526-096c-45de-a569-4e80570d9b28-kube-api-access-r6bnj\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.644229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.643971 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.644229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.644193 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fef5526-096c-45de-a569-4e80570d9b28-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.644442 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.644285 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-metrics-certs\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.645951 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.645899 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fef5526-096c-45de-a569-4e80570d9b28-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.648658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.648598 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-metrics-certs\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.649043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.648890 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-maas-api-tls\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.668633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.668562 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6bnj\" (UniqueName: \"kubernetes.io/projected/8fef5526-096c-45de-a569-4e80570d9b28-kube-api-access-r6bnj\") pod \"maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.725085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.724733 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:31.777633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.777535 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:31.922920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:31.922825 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:32.027885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:32.027703 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:32.133331 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:32.133252 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9"] Oct 30 18:10:32.135675 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:10:32.135586 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fef5526_096c_45de_a569_4e80570d9b28.slice/crio-eb731e1108bfb49d91497255711ba1c7f4efd7532226986148949ace568e8fde WatchSource:0}: Error finding container eb731e1108bfb49d91497255711ba1c7f4efd7532226986148949ace568e8fde: Status 404 returned error can't find the container with id eb731e1108bfb49d91497255711ba1c7f4efd7532226986148949ace568e8fde Oct 30 18:10:32.751007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:32.750846 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" event={"ID":"8fef5526-096c-45de-a569-4e80570d9b28","Type":"ContainerStarted","Data":"eb731e1108bfb49d91497255711ba1c7f4efd7532226986148949ace568e8fde"} Oct 30 18:10:32.890193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:32.890122 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:33.007824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.007737 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:33.077608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.077519 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt"] Oct 30 18:10:33.083745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.083692 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:33.103526 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.103447 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt"] Oct 30 18:10:33.108906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.108839 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8b6cf1\"" Oct 30 18:10:33.108906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.108842 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8b6cf1-dockercfg-mmwdg\"" Oct 30 18:10:33.186066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.177639 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d866fec-8541-40d6-ae37-11dcfef32e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt\" (UID: \"9d866fec-8541-40d6-ae37-11dcfef32e46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:33.186066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.178074 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/9d866fec-8541-40d6-ae37-11dcfef32e46-kube-api-access-r77qb\") pod \"payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt\" (UID: \"9d866fec-8541-40d6-ae37-11dcfef32e46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:33.282329 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.281865 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d866fec-8541-40d6-ae37-11dcfef32e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt\" (UID: \"9d866fec-8541-40d6-ae37-11dcfef32e46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:33.282329 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.282233 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/9d866fec-8541-40d6-ae37-11dcfef32e46-kube-api-access-r77qb\") pod \"payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt\" (UID: \"9d866fec-8541-40d6-ae37-11dcfef32e46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:33.285806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.284767 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d866fec-8541-40d6-ae37-11dcfef32e46-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt\" (UID: \"9d866fec-8541-40d6-ae37-11dcfef32e46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:33.314659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.314603 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/9d866fec-8541-40d6-ae37-11dcfef32e46-kube-api-access-r77qb\") pod \"payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt\" (UID: \"9d866fec-8541-40d6-ae37-11dcfef32e46\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:33.426056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.425983 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:33.508218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.508155 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:33.608496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.608411 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:33.767488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.767308 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" event={"ID":"8fef5526-096c-45de-a569-4e80570d9b28","Type":"ContainerStarted","Data":"12cd149cb0e8c60f6acd70ea41fa2152bc05a67eabc821da31e0c30195906880"} Oct 30 18:10:33.767488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.767437 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:33.808835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.808760 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt"] Oct 30 18:10:33.811440 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:10:33.811367 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d866fec_8541_40d6_ae37_11dcfef32e46.slice/crio-c6bab1b085255d69b7be81c031236b631e7b62e14738dc6d5b1e5bfe43868070 WatchSource:0}: Error finding container c6bab1b085255d69b7be81c031236b631e7b62e14738dc6d5b1e5bfe43868070: Status 404 returned error can't find the container with id c6bab1b085255d69b7be81c031236b631e7b62e14738dc6d5b1e5bfe43868070 Oct 30 18:10:33.817407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:33.817290 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" podStartSLOduration=2.383506349 podStartE2EDuration="2.817249271s" totalImagesPullingTime="433.742922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:33.814138906 +0000 UTC m=+528.202667025" watchObservedRunningTime="2026-10-30 18:10:33.817249271 +0000 UTC m=+528.205777391" Oct 30 18:10:34.782388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:34.782305 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" event={"ID":"9d866fec-8541-40d6-ae37-11dcfef32e46","Type":"ContainerStarted","Data":"fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1"} Oct 30 18:10:34.783259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:34.782445 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" event={"ID":"9d866fec-8541-40d6-ae37-11dcfef32e46","Type":"ContainerStarted","Data":"c6bab1b085255d69b7be81c031236b631e7b62e14738dc6d5b1e5bfe43868070"} Oct 30 18:10:34.783259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:34.782824 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:34.838471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:34.838324 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" podStartSLOduration=1.838281984 podStartE2EDuration="1.838281984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:34.829628851 +0000 UTC m=+529.218156963" watchObservedRunningTime="2026-10-30 18:10:34.838281984 +0000 UTC m=+529.226810102" Oct 30 18:10:35.214322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.214242 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:35.307469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.307353 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:35.407963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.407875 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:35.563058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.562890 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:35.667303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.667226 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:35.767096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.767033 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht"] Oct 30 18:10:35.804203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.804069 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht"] Oct 30 18:10:35.805121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.804235 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:35.859073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.853177 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6ce5eb\"" Oct 30 18:10:35.865480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.865271 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6ce5eb-dockercfg-hs7vt\"" Oct 30 18:10:35.967737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.964492 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff"] Oct 30 18:10:35.972263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.972047 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx6k8\" (UniqueName: \"kubernetes.io/projected/69c13883-2b83-47a6-a8c1-39ae5cc32976-kube-api-access-jx6k8\") pod \"payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht\" (UID: \"69c13883-2b83-47a6-a8c1-39ae5cc32976\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:35.972744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.972531 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c13883-2b83-47a6-a8c1-39ae5cc32976-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht\" (UID: \"69c13883-2b83-47a6-a8c1-39ae5cc32976\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:35.984244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.983993 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff"] Oct 30 18:10:35.986570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:35.984145 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:36.074849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.074765 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c13883-2b83-47a6-a8c1-39ae5cc32976-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht\" (UID: \"69c13883-2b83-47a6-a8c1-39ae5cc32976\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:36.075189 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.074962 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx6k8\" (UniqueName: \"kubernetes.io/projected/69c13883-2b83-47a6-a8c1-39ae5cc32976-kube-api-access-jx6k8\") pod \"payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht\" (UID: \"69c13883-2b83-47a6-a8c1-39ae5cc32976\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:36.076045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.076016 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c13883-2b83-47a6-a8c1-39ae5cc32976-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht\" (UID: \"69c13883-2b83-47a6-a8c1-39ae5cc32976\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:36.098045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.097991 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx6k8\" (UniqueName: \"kubernetes.io/projected/69c13883-2b83-47a6-a8c1-39ae5cc32976-kube-api-access-jx6k8\") pod \"payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht\" (UID: \"69c13883-2b83-47a6-a8c1-39ae5cc32976\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:36.177054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.176974 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqbn\" (UniqueName: \"kubernetes.io/projected/f339d950-7963-41f3-99ea-237bca9464dd-kube-api-access-6rqbn\") pod \"payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff\" (UID: \"f339d950-7963-41f3-99ea-237bca9464dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:36.177371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.177158 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f339d950-7963-41f3-99ea-237bca9464dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff\" (UID: \"f339d950-7963-41f3-99ea-237bca9464dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:36.310043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.291348 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f339d950-7963-41f3-99ea-237bca9464dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff\" (UID: \"f339d950-7963-41f3-99ea-237bca9464dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:36.310043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.291777 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqbn\" (UniqueName: \"kubernetes.io/projected/f339d950-7963-41f3-99ea-237bca9464dd-kube-api-access-6rqbn\") pod \"payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff\" (UID: \"f339d950-7963-41f3-99ea-237bca9464dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:36.310043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.298833 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f339d950-7963-41f3-99ea-237bca9464dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff\" (UID: \"f339d950-7963-41f3-99ea-237bca9464dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:36.310043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.301449 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:36.324778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.324155 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqbn\" (UniqueName: \"kubernetes.io/projected/f339d950-7963-41f3-99ea-237bca9464dd-kube-api-access-6rqbn\") pod \"payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff\" (UID: \"f339d950-7963-41f3-99ea-237bca9464dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:36.340574 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.340515 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:36.576914 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.576849 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:36.728603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.728529 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht"] Oct 30 18:10:36.730449 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:10:36.730379 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c13883_2b83_47a6_a8c1_39ae5cc32976.slice/crio-cf1b5bfb2c3a3ff89f093e13e4369b38c8bae50bbae63691281898cd857709ab WatchSource:0}: Error finding container cf1b5bfb2c3a3ff89f093e13e4369b38c8bae50bbae63691281898cd857709ab: Status 404 returned error can't find the container with id cf1b5bfb2c3a3ff89f093e13e4369b38c8bae50bbae63691281898cd857709ab Oct 30 18:10:36.750079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.749901 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff"] Oct 30 18:10:36.757627 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:10:36.757564 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf339d950_7963_41f3_99ea_237bca9464dd.slice/crio-278893e636aab10868fd388cb811f1349bf71c6e5431bb0d369eb57a833e8cf3 WatchSource:0}: Error finding container 278893e636aab10868fd388cb811f1349bf71c6e5431bb0d369eb57a833e8cf3: Status 404 returned error can't find the container with id 278893e636aab10868fd388cb811f1349bf71c6e5431bb0d369eb57a833e8cf3 Oct 30 18:10:36.762915 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.762607 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:36.837339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.835603 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" event={"ID":"69c13883-2b83-47a6-a8c1-39ae5cc32976","Type":"ContainerStarted","Data":"cf1b5bfb2c3a3ff89f093e13e4369b38c8bae50bbae63691281898cd857709ab"} Oct 30 18:10:36.842178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:36.842102 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" event={"ID":"f339d950-7963-41f3-99ea-237bca9464dd","Type":"ContainerStarted","Data":"278893e636aab10868fd388cb811f1349bf71c6e5431bb0d369eb57a833e8cf3"} Oct 30 18:10:37.077054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:37.069525 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:37.109011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:37.107592 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:10:37.857069 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:37.856997 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" event={"ID":"69c13883-2b83-47a6-a8c1-39ae5cc32976","Type":"ContainerStarted","Data":"3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7"} Oct 30 18:10:37.857870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:37.857171 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:37.861098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:37.861019 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" event={"ID":"f339d950-7963-41f3-99ea-237bca9464dd","Type":"ContainerStarted","Data":"cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3"} Oct 30 18:10:37.861430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:37.861138 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:37.905712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:37.905588 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" podStartSLOduration=2.905551098 podStartE2EDuration="2.905551098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:37.901590812 +0000 UTC m=+532.290118929" watchObservedRunningTime="2026-10-30 18:10:37.905551098 +0000 UTC m=+532.294079215" Oct 30 18:10:37.948254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:37.947922 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" podStartSLOduration=2.9478865340000002 podStartE2EDuration="2.947886534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:37.945230332 +0000 UTC m=+532.333758610" watchObservedRunningTime="2026-10-30 18:10:37.947886534 +0000 UTC m=+532.336414658" Oct 30 18:10:38.679271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:38.679198 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:38.811032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:38.808141 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:38.878077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:38.877737 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq"] Oct 30 18:10:38.933780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:38.933574 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:38.947455 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:38.947143 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:38.949959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:38.949840 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq"] Oct 30 18:10:39.038018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.029637 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4ef6ac\"" Oct 30 18:10:39.064044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.039877 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4ef6ac-dockercfg-rhmzx\"" Oct 30 18:10:39.160973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.160854 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:39.249771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.249554 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnms\" (UniqueName: \"kubernetes.io/projected/5619b5d8-1160-498f-bcd3-990f8bf5def5-kube-api-access-htnms\") pod \"payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq\" (UID: \"5619b5d8-1160-498f-bcd3-990f8bf5def5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:39.249771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.249734 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619b5d8-1160-498f-bcd3-990f8bf5def5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq\" (UID: \"5619b5d8-1160-498f-bcd3-990f8bf5def5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:39.351425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.351351 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnms\" (UniqueName: \"kubernetes.io/projected/5619b5d8-1160-498f-bcd3-990f8bf5def5-kube-api-access-htnms\") pod \"payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq\" (UID: \"5619b5d8-1160-498f-bcd3-990f8bf5def5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:39.351808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.351560 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619b5d8-1160-498f-bcd3-990f8bf5def5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq\" (UID: \"5619b5d8-1160-498f-bcd3-990f8bf5def5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:39.352867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.352811 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619b5d8-1160-498f-bcd3-990f8bf5def5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq\" (UID: \"5619b5d8-1160-498f-bcd3-990f8bf5def5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:39.373851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.373783 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnms\" (UniqueName: \"kubernetes.io/projected/5619b5d8-1160-498f-bcd3-990f8bf5def5-kube-api-access-htnms\") pod \"payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq\" (UID: \"5619b5d8-1160-498f-bcd3-990f8bf5def5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:39.476240 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.476153 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:39.557342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.557258 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:39.824542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.824385 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:10:39.969408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:39.962562 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq"] Oct 30 18:10:40.577664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.577567 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2"] Oct 30 18:10:40.583560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.583499 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.609043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.608877 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6ce5eb-serving-cert\"" Oct 30 18:10:40.628382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.628319 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2"] Oct 30 18:10:40.660294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.660225 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:40.683726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.683007 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp2c8\" (UniqueName: \"kubernetes.io/projected/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-kube-api-access-zp2c8\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.683726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.683103 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.683726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.683167 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.683726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.683214 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.790990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.785325 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.790990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.785459 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.790990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.785521 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.790990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.785896 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp2c8\" (UniqueName: \"kubernetes.io/projected/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-kube-api-access-zp2c8\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.814765 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.814684 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.815203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.815149 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:40.821532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.821452 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-metrics-certs\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.822586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.822513 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.838470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.838300 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp2c8\" (UniqueName: \"kubernetes.io/projected/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-kube-api-access-zp2c8\") pod \"maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.942998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.942176 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:40.952184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.952122 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:40.976172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.976103 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" event={"ID":"5619b5d8-1160-498f-bcd3-990f8bf5def5","Type":"ContainerStarted","Data":"3850f5f1b274436d287689ba50b5652b1312d6fa3cce98728a2aa53807833a8a"} Oct 30 18:10:40.976888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.976228 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" event={"ID":"5619b5d8-1160-498f-bcd3-990f8bf5def5","Type":"ContainerStarted","Data":"a4bf0c1cd4185c017dbe880359e62246544701c6105eb55bbdd1d17984fc72e9"} Oct 30 18:10:40.976888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:40.976443 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:41.043629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:41.043471 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" podStartSLOduration=3.043427456 podStartE2EDuration="3.043427456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:41.034103116 +0000 UTC m=+535.422631233" watchObservedRunningTime="2026-10-30 18:10:41.043427456 +0000 UTC m=+535.431955572" Oct 30 18:10:41.362766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:41.362686 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2"] Oct 30 18:10:41.366569 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:10:41.366508 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56b6bd9_b1d9_483f_92f8_8a4aadad34ac.slice/crio-a2a51eab216277c95eb754d6d9565c5e78a6c240c7259e0f37ba53687cc0018d WatchSource:0}: Error finding container a2a51eab216277c95eb754d6d9565c5e78a6c240c7259e0f37ba53687cc0018d: Status 404 returned error can't find the container with id a2a51eab216277c95eb754d6d9565c5e78a6c240c7259e0f37ba53687cc0018d Oct 30 18:10:42.032026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:42.031891 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" event={"ID":"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac","Type":"ContainerStarted","Data":"a2a51eab216277c95eb754d6d9565c5e78a6c240c7259e0f37ba53687cc0018d"} Oct 30 18:10:42.443976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:42.443263 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:42.540706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:42.540467 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:42.641909 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:42.641830 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:43.049560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:43.049447 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" event={"ID":"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac","Type":"ContainerStarted","Data":"fd2344d3f2e79849a9bff032e82460446eb04ca6cf2132bb824cae4989d29ce4"} Oct 30 18:10:43.050217 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:43.049678 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:44.169699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:44.169547 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:45.949037 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:45.947313 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:46.070778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:46.070684 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:10:46.079386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:46.078655 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:46.248997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:46.231450 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" podStartSLOduration=5.722808102 podStartE2EDuration="6.23140479s" totalImagesPullingTime="508.596688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:43.10364853 +0000 UTC m=+537.492176648" watchObservedRunningTime="2026-10-30 18:10:46.23140479 +0000 UTC m=+540.619932908" Oct 30 18:10:46.387626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:46.382699 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:46.657222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:46.643355 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:47.050827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:47.050482 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:47.180983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:47.177846 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:47.347299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:47.344827 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:47.959025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:47.958851 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:48.673718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:48.673616 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:48.961787 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:48.959415 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:10:48.971516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:48.971393 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:10:49.176048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:49.176006 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:10:50.140795 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:10:50.140603 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:10:50.169622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:50.169553 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:50.997989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:50.982378 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:10:52.050009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:52.048764 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:10:52.165120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:52.164233 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:52.368675 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:52.368599 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:53.018985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:53.017926 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:53.122400 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:53.122171 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:53.241384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:53.240035 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:54.368346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:54.368277 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:56.396546 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:56.396466 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:56.697432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:56.697225 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:56.702977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:56.697902 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:57.550589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:57.550494 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:57.655806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:57.655023 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:57.655806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:57.655150 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:57.659998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:57.659798 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:57.703669 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:57.703570 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:10:57.704377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:57.704299 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:10:58.191757 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:58.191658 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:58.330543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:58.330345 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:58.464826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:58.464650 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:58.751066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:10:58.743319 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:00.572286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:00.572191 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:02.249091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:02.249014 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:02.924208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:02.924117 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:03.035721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:03.035632 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:03.570672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:03.570604 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:03.771361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:03.771299 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:04.213443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:04.041457 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:04.213443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:04.169099 5624 generic.go:356] "Generic (PLEG): container finished" podID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerID="e6c0619e00ae7c0869b255b050920b47562d7bf2e5f0ca002d42bdebb0acd96e" exitCode=0 Oct 30 18:11:04.213443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:04.169189 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" event={"ID":"ac309921-c2c5-4c6a-8635-79dea80cfadf","Type":"ContainerDied","Data":"e6c0619e00ae7c0869b255b050920b47562d7bf2e5f0ca002d42bdebb0acd96e"} Oct 30 18:11:04.213443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:04.170144 5624 scope.go:122] "RemoveContainer" containerID="e6c0619e00ae7c0869b255b050920b47562d7bf2e5f0ca002d42bdebb0acd96e" Oct 30 18:11:04.213443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:04.173663 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:04.213443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:04.173826 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:04.213443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:04.173889 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:04.213443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:04.183904 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:05.190097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.190038 5624 generic.go:356] "Generic (PLEG): container finished" podID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerID="54f17b7248f7563e7f291f5874d4a5b435a9dcfe1dc08af6b6723bc42f9c38b9" exitCode=0 Oct 30 18:11:05.190896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.190149 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" event={"ID":"56a9e445-860a-46c5-80bd-6ed297d09b77","Type":"ContainerDied","Data":"54f17b7248f7563e7f291f5874d4a5b435a9dcfe1dc08af6b6723bc42f9c38b9"} Oct 30 18:11:05.191630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.191579 5624 scope.go:122] "RemoveContainer" containerID="54f17b7248f7563e7f291f5874d4a5b435a9dcfe1dc08af6b6723bc42f9c38b9" Oct 30 18:11:05.200332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.200201 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:05.200717 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.200438 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:05.200717 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.200507 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:05.205377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.205173 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" event={"ID":"ac309921-c2c5-4c6a-8635-79dea80cfadf","Type":"ContainerStarted","Data":"ad8e94c5a5bf4cd3d35a8732a7cf746502d07a2f00af6478e9d64fb34ef121ca"} Oct 30 18:11:05.602841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.602613 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:11:05.711731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.701595 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:05.711731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.701742 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:11:05.728397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:05.712086 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:11:06.227987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.215876 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:06.278019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.273615 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" event={"ID":"56a9e445-860a-46c5-80bd-6ed297d09b77","Type":"ContainerStarted","Data":"f50c670664203c625a5333421eeaf48fffec42081705a541a634e9a70d9fe6e9"} Oct 30 18:11:06.317173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.307318 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:06.317173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.307444 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:06.317173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.315615 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:11:06.397059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.364485 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerID="82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae" exitCode=0 Oct 30 18:11:06.397059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.375959 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" event={"ID":"cc4d072e-56c8-4f49-966c-18445c91eea2","Type":"ContainerDied","Data":"82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae"} Oct 30 18:11:06.449533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.449456 5624 scope.go:122] "RemoveContainer" containerID="82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae" Oct 30 18:11:06.459673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.459567 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:06.477631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.477551 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:06.487311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.487092 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:06.766246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.766077 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:06.766246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:06.766212 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:11:07.299353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:07.295885 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:07.299353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:07.296069 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:07.455693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:07.455603 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" event={"ID":"cc4d072e-56c8-4f49-966c-18445c91eea2","Type":"ContainerStarted","Data":"897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d"} Oct 30 18:11:07.703987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:07.703847 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:07.706821 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:07.705984 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:11:07.881971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:07.881088 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:11:08.220983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:08.220733 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:08.322977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:08.322057 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:08.322977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:08.322230 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:08.789986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:08.780301 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:08.789986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:08.780502 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:11:08.873987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:08.871027 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:09.075137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.055180 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:09.075137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.055312 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:11:09.334033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.333797 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:09.364280 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.364135 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:09.405868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.405807 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2"] Oct 30 18:11:09.492998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.459837 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:09.599330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.579282 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:09.626214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.607503 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:09.626214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.607696 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:09.626214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.607804 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:09.626214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.607859 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:09.726030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.678760 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2"] Oct 30 18:11:09.821990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.821344 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:09.821990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.821473 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:11:09.910088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.857874 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:09.910088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.858029 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:11:09.974986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:09.955817 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldlt\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-kube-api-access-bldlt\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.025229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.022129 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.025229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.022439 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.025229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.022517 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.025229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.022616 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8824e58-efac-475b-8dbe-3f105d4d2f32-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.207103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.193372 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:10.336987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.312063 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:10.368450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.368345 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:10.498013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.436822 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:10.498013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.486450 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.533988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.487256 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.557001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.556492 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.575991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.573172 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.575991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.573293 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.575991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.573369 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldlt\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-kube-api-access-bldlt\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.575991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.573412 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.575991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.573657 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.575991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.573749 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.575991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.573835 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8824e58-efac-475b-8dbe-3f105d4d2f32-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.612018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.586155 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.641322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.641228 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:11:10.687986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.687499 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.711983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.693056 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.711983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.697390 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.772171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.763538 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8824e58-efac-475b-8dbe-3f105d4d2f32-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.772171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.763774 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:10.772171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.763864 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:11:10.772171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.769414 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.840277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.839078 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:10.840277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.839205 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:11:10.874972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.874845 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:10.902694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:10.885430 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldlt\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-kube-api-access-bldlt\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:11.068907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.068665 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:11.328200 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.322432 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:11.328200 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.322543 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:11.659234 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.659157 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:11:11.717983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.697796 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:11:11.873993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.873796 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:11.897903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.897780 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:11:11.952160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.943530 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:11.952160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.945741 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:11:11.966042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:11.953628 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:12.512624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:12.512535 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:12.529046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:12.527310 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:12.744242 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:12.740843 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:12.902704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:12.886825 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:11:12.902704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:12.887002 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:11:12.902704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:12.900208 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:12.902704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:12.900361 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:11:13.265418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.265338 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:13.281206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.273209 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:13.323365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.323142 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2"] Oct 30 18:11:13.369044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.366821 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:13.369044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.366968 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:13.582815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.581183 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:13.582815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.581424 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:13.582815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.581507 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:13.582815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.581715 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" event={"ID":"b8824e58-efac-475b-8dbe-3f105d4d2f32","Type":"ContainerStarted","Data":"5ca0ea53b75d2490ea95c99b749b58a4ed000925b1882ce3b40beedffc3580f4"} Oct 30 18:11:13.676578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.676381 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:13.823219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.821765 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:13.823219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:13.821885 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:11:14.296363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:14.295054 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:11:14.296363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:14.295185 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:11:14.675556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:14.675448 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" event={"ID":"b8824e58-efac-475b-8dbe-3f105d4d2f32","Type":"ContainerStarted","Data":"a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1"} Oct 30 18:11:14.764056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:14.763981 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:11:14.764566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:14.764507 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:11:14.776387 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:14.776260 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podStartSLOduration=6.776216377 podStartE2EDuration="6.776216377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:14.753015514 +0000 UTC m=+569.141543623" watchObservedRunningTime="2026-10-30 18:11:14.776216377 +0000 UTC m=+569.164744495" Oct 30 18:11:14.823025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:14.822916 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:14.825770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:14.825668 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:11:15.070892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.070525 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:15.074352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.074255 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:11:15.074737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.074397 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:11:15.077316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.077245 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:11:15.170887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.170814 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:11:15.389294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.387074 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:11:15.393008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.391704 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:15.472020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.442459 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:11:15.823085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.822020 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:11:15.823085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:15.822226 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:11:16.071089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:16.071022 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:11:16.072059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:16.071106 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:11:16.295114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:16.295006 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:16.830706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:16.830612 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:11:16.833665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:16.833592 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:11:17.058868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.058769 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:17.070735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.070645 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:11:17.071176 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.070804 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:11:17.731022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.730145 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:17.747163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.746858 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2"] Oct 30 18:11:17.771179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.771097 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2"] Oct 30 18:11:17.771539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.771353 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.821073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.821011 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"simulator-w6-ef13cc-kserve-self-signed-certs\"" Oct 30 18:11:17.822522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.822449 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-dshm\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.822818 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.822538 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwhq\" (UniqueName: \"kubernetes.io/projected/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-kube-api-access-fcwhq\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.822818 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.822665 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-home\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.823095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.822834 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-model-cache\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.823095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.822868 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tls-certs\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.823229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.823163 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tmp-dir\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.833099 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.832927 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"kube-root-ca.crt\"" Oct 30 18:11:17.833454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.833301 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"default-dockercfg-45pmf\"" Oct 30 18:11:17.833527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.833460 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"openshift-service-ca.crt\"" Oct 30 18:11:17.943172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.928505 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-model-cache\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.955361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.943699 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tls-certs\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.955361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.944089 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tmp-dir\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.955361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.944271 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-dshm\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.955361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.944308 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwhq\" (UniqueName: \"kubernetes.io/projected/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-kube-api-access-fcwhq\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.955361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.954038 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-model-cache\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.962376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.962281 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-home\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.969833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.968839 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tmp-dir\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.970254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.969913 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-home\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.974716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.974655 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-dshm\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:17.977274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:17.977211 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tls-certs\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:18.035161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.029137 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwhq\" (UniqueName: \"kubernetes.io/projected/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-kube-api-access-fcwhq\") pod \"simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:18.070954 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.070853 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:11:18.071426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.071114 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:11:18.139218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.139128 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:18.549297 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:11:18.549219 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16d4b0da_7d2b_49c3_b2c3_9c450f7a79a8.slice/crio-c48e993f89c3ddf3f32da5d19c3032e653b2dd47e72e39339c37da4132490d7e WatchSource:0}: Error finding container c48e993f89c3ddf3f32da5d19c3032e653b2dd47e72e39339c37da4132490d7e: Status 404 returned error can't find the container with id c48e993f89c3ddf3f32da5d19c3032e653b2dd47e72e39339c37da4132490d7e Oct 30 18:11:18.549616 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.549314 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2"] Oct 30 18:11:18.753519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.753402 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:18.875787 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.875692 5624 generic.go:356] "Generic (PLEG): container finished" podID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerID="0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52" exitCode=0 Oct 30 18:11:18.876226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.876141 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" event={"ID":"df5293e2-5b63-430f-aa36-9733a6a35dd3","Type":"ContainerDied","Data":"0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52"} Oct 30 18:11:18.877472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.877413 5624 scope.go:122] "RemoveContainer" containerID="0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52" Oct 30 18:11:18.885800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.883124 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:18.885800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.883419 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:18.885800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.883529 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:18.911379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.898902 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" event={"ID":"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8","Type":"ContainerStarted","Data":"306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030"} Oct 30 18:11:18.911379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:18.899261 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" event={"ID":"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8","Type":"ContainerStarted","Data":"c48e993f89c3ddf3f32da5d19c3032e653b2dd47e72e39339c37da4132490d7e"} Oct 30 18:11:19.070632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:19.070551 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:11:19.071063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:19.070691 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:11:19.149786 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:19.147732 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:19.934619 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:19.934516 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" event={"ID":"df5293e2-5b63-430f-aa36-9733a6a35dd3","Type":"ContainerStarted","Data":"9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a"} Oct 30 18:11:19.974110 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:11:19.974041 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:11:20.021768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:20.021612 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podStartSLOduration=3.021560605 podStartE2EDuration="3.021560605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:18.993807477 +0000 UTC m=+573.382335597" watchObservedRunningTime="2026-10-30 18:11:20.021560605 +0000 UTC m=+574.410088723" Oct 30 18:11:20.070515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:20.070435 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:11:20.070899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:20.070546 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:11:20.151987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:20.151581 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:11:20.153948 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:20.153861 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:20.154273 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:20.154070 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:11:20.447013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:20.438485 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:21.071075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:21.071002 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:11:21.071418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:21.071163 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:11:21.150992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:21.150873 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:21.151366 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:21.151014 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:11:21.777810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:21.777727 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:11:22.051106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:22.048121 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:22.070524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:22.070445 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:11:22.070987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:22.070552 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:11:22.157057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:22.152829 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:22.157057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:22.152983 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:11:23.198998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:23.182322 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:23.198998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:23.182439 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:11:23.247692 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:23.247627 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:23.260716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:23.257069 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:23.264586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:23.264516 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:11:23.749989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:23.745201 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:24.151608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:24.151526 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:24.152041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:24.151671 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:11:25.152025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:25.151892 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:25.152714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:25.152059 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:11:25.237675 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:25.237584 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:26.151718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.151653 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:26.152131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.151761 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:11:26.651180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.651113 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:11:26.727294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.727210 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d"] Oct 30 18:11:26.750157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.747911 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d"] Oct 30 18:11:26.750157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.748118 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.760250 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.760073 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:26.776597 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.776524 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19508c5-5ea0-4090-85b3-db567fa6dd86-tls-certs\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.777071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.776723 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfnf\" (UniqueName: \"kubernetes.io/projected/d19508c5-5ea0-4090-85b3-db567fa6dd86-kube-api-access-slfnf\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.777071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.776768 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-dshm\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.777071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.776973 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-home\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.777071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.777005 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-tmp-dir\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.777352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.777070 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-model-cache\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.778121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.778074 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"openshift-service-ca.crt\"" Oct 30 18:11:26.778657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.778246 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"default-dockercfg-d99qs\"" Oct 30 18:11:26.779024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.778380 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"kube-root-ca.crt\"" Oct 30 18:11:26.779222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.778463 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"premium-w2-fde2c5-kserve-self-signed-certs\"" Oct 30 18:11:26.863632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.863563 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:26.879569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.879485 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-home\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.879569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.879555 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-tmp-dir\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.880060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.879616 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-model-cache\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.880060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.879807 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19508c5-5ea0-4090-85b3-db567fa6dd86-tls-certs\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.880060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.880035 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfnf\" (UniqueName: \"kubernetes.io/projected/d19508c5-5ea0-4090-85b3-db567fa6dd86-kube-api-access-slfnf\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.880246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.880068 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-dshm\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.880347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.880279 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-home\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.880544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.880408 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-tmp-dir\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.881599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.881526 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-model-cache\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.899186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.892164 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19508c5-5ea0-4090-85b3-db567fa6dd86-tls-certs\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.949078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.936311 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfnf\" (UniqueName: \"kubernetes.io/projected/d19508c5-5ea0-4090-85b3-db567fa6dd86-kube-api-access-slfnf\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:26.976984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:26.976575 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-dshm\") pod \"premium-w2-fde2c5-kserve-554b5bc78c-lt54d\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:27.081270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:27.081178 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:27.153458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:27.152707 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:11:27.153458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:27.152870 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:11:27.499349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:27.499244 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d"] Oct 30 18:11:27.500344 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:11:27.500281 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd19508c5_5ea0_4090_85b3_db567fa6dd86.slice/crio-7a19dbd3906a46d2f87436c237d460d594ff9123a9a348afab16104d5b91355c WatchSource:0}: Error finding container 7a19dbd3906a46d2f87436c237d460d594ff9123a9a348afab16104d5b91355c: Status 404 returned error can't find the container with id 7a19dbd3906a46d2f87436c237d460d594ff9123a9a348afab16104d5b91355c Oct 30 18:11:28.107612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:28.107526 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" event={"ID":"d19508c5-5ea0-4090-85b3-db567fa6dd86","Type":"ContainerStarted","Data":"ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12"} Oct 30 18:11:28.107612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:28.107624 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" event={"ID":"d19508c5-5ea0-4090-85b3-db567fa6dd86","Type":"ContainerStarted","Data":"7a19dbd3906a46d2f87436c237d460d594ff9123a9a348afab16104d5b91355c"} Oct 30 18:11:28.139869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:28.139787 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:28.150625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:28.150490 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podStartSLOduration=2.150456669 podStartE2EDuration="2.150456669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:28.148567274 +0000 UTC m=+582.537095395" watchObservedRunningTime="2026-10-30 18:11:28.150456669 +0000 UTC m=+582.538984780" Oct 30 18:11:28.157261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:28.157193 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:28.162054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:28.161914 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:11:28.164610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:28.164555 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:11:28.534054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:28.533980 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:29.082201 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:29.082143 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:29.130953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:29.130858 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:11:29.723028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:29.722946 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:30.064710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:30.064525 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:31.634888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:31.634809 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:31.747163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:31.747080 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:33.124722 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:33.124624 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:34.847710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:34.847616 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:36.518114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:36.518054 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:37.082249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:37.082173 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:37.099271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:37.099195 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:37.238405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:37.238342 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:11:37.648715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:37.648625 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:38.144527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:38.144457 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:39.677108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:39.677057 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:39.830489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:39.830431 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:40.562398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:40.562300 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:11:41.433466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:41.433394 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:41.606589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:41.606448 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:43.113563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:43.113478 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:44.722240 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:44.722159 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:46.264117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:46.263929 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:47.955098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:47.954994 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:49.444562 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:49.444484 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:49.648681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:49.648240 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:49.941407 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:11:49.941315 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:11:51.267144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:51.267081 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:52.832374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:52.832303 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:53.027715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:53.027639 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:54.048927 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.048857 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9"] Oct 30 18:11:54.057609 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.057548 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.081706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.081412 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9"] Oct 30 18:11:54.137023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.136920 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm"] Oct 30 18:11:54.143700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.143617 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.144516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.144451 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.144753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.144559 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.144753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.144604 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xm7p\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-kube-api-access-6xm7p\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.144753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.144676 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.145057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.144913 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.145157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.145065 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.145157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.145108 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.145157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.145138 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadedad1-fea9-471c-bd58-2372d6c91bb8-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.148310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.147115 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.150223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.150090 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm"] Oct 30 18:11:54.167533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.167473 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:54.168033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.167643 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"openshift-service-ca.crt\"" Oct 30 18:11:54.168033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.167772 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"kube-root-ca.crt\"" Oct 30 18:11:54.168033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.167771 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"premium-w0-6ce5eb-kserve-self-signed-certs\"" Oct 30 18:11:54.168033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.167905 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"default-dockercfg-j6p62\"" Oct 30 18:11:54.246652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.246560 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.246652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.246646 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.246686 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadedad1-fea9-471c-bd58-2372d6c91bb8-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.246745 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pmqn\" (UniqueName: \"kubernetes.io/projected/ccc1e591-0916-4cad-85a0-10091e148e43-kube-api-access-2pmqn\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.247146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.246842 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247107 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccc1e591-0916-4cad-85a0-10091e148e43-tls-certs\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.247383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247249 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247265 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247387 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247431 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xm7p\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-kube-api-access-6xm7p\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247446 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247470 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-home\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.247704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247505 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-dshm\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.247704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247609 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-tmp-dir\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.247809 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247724 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247809 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247764 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.247928 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.247910 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadedad1-fea9-471c-bd58-2372d6c91bb8-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.248108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.248086 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.248180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.248120 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-model-cache\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.248472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.248434 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.251189 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.251152 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.251641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.251607 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.270557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.270494 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.270872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.270815 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xm7p\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-kube-api-access-6xm7p\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.355088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.353239 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccc1e591-0916-4cad-85a0-10091e148e43-tls-certs\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.355088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.353371 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-home\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.355088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.353399 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-dshm\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.355088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.353437 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-tmp-dir\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.355088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.353638 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-model-cache\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.355088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.353795 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pmqn\" (UniqueName: \"kubernetes.io/projected/ccc1e591-0916-4cad-85a0-10091e148e43-kube-api-access-2pmqn\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.355088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.354328 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-home\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.356992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.356133 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-model-cache\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.356992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.354921 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-tmp-dir\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.359497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.359438 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccc1e591-0916-4cad-85a0-10091e148e43-tls-certs\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.365635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.365469 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-dshm\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.401687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.400549 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pmqn\" (UniqueName: \"kubernetes.io/projected/ccc1e591-0916-4cad-85a0-10091e148e43-kube-api-access-2pmqn\") pod \"premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.401687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.401167 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:54.480164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.480086 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:54.571554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.571470 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:54.802296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.802205 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9"] Oct 30 18:11:54.811325 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:11:54.811232 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadedad1_fea9_471c_bd58_2372d6c91bb8.slice/crio-819b1c9e1f2935b1a2dfa6ddea97b7ff361adbe0e5bd0564381732850f8e8926 WatchSource:0}: Error finding container 819b1c9e1f2935b1a2dfa6ddea97b7ff361adbe0e5bd0564381732850f8e8926: Status 404 returned error can't find the container with id 819b1c9e1f2935b1a2dfa6ddea97b7ff361adbe0e5bd0564381732850f8e8926 Oct 30 18:11:54.817604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.817493 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:54.818056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.817711 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:54.818056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.817784 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:11:54.960761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:54.960679 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm"] Oct 30 18:11:54.964175 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:11:54.964108 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccc1e591_0916_4cad_85a0_10091e148e43.slice/crio-497db115fe2ca6d927d1eac383b4a963df6dd244faad4d27c6be0e0d0c726255 WatchSource:0}: Error finding container 497db115fe2ca6d927d1eac383b4a963df6dd244faad4d27c6be0e0d0c726255: Status 404 returned error can't find the container with id 497db115fe2ca6d927d1eac383b4a963df6dd244faad4d27c6be0e0d0c726255 Oct 30 18:11:55.593568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:55.593483 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" event={"ID":"aadedad1-fea9-471c-bd58-2372d6c91bb8","Type":"ContainerStarted","Data":"263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b"} Oct 30 18:11:55.594558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:55.593605 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" event={"ID":"aadedad1-fea9-471c-bd58-2372d6c91bb8","Type":"ContainerStarted","Data":"819b1c9e1f2935b1a2dfa6ddea97b7ff361adbe0e5bd0564381732850f8e8926"} Oct 30 18:11:55.598062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:55.597998 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" event={"ID":"ccc1e591-0916-4cad-85a0-10091e148e43","Type":"ContainerStarted","Data":"88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad"} Oct 30 18:11:55.598062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:55.598070 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" event={"ID":"ccc1e591-0916-4cad-85a0-10091e148e43","Type":"ContainerStarted","Data":"497db115fe2ca6d927d1eac383b4a963df6dd244faad4d27c6be0e0d0c726255"} Oct 30 18:11:55.643118 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:55.643002 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podStartSLOduration=1.6429635870000001 podStartE2EDuration="1.642963587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:55.638236469 +0000 UTC m=+610.026764585" watchObservedRunningTime="2026-10-30 18:11:55.642963587 +0000 UTC m=+610.031491695" Oct 30 18:11:55.679744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:55.679625 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podStartSLOduration=1.6795884810000001 podStartE2EDuration="1.679588481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:55.678143503 +0000 UTC m=+610.066671620" watchObservedRunningTime="2026-10-30 18:11:55.679588481 +0000 UTC m=+610.068116599" Oct 30 18:11:56.410983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:56.409073 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:11:56.424710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:56.424646 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:11:56.425215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:56.425167 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:11:56.456534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:56.456458 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:56.480649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:56.480580 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:11:57.173556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:57.173487 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:57.403990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:57.403394 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:11:57.403990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:57.403547 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:11:57.414012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:57.413318 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:11:58.408981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:58.406372 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:11:58.408981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:58.406532 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:11:58.842080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:58.841490 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:59.406992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:59.404987 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:11:59.406992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:59.405109 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:11:59.643984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:11:59.606918 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:00.445836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:00.442036 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:12:00.445836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:00.442182 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:12:00.994243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:00.994154 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:01.416980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:01.404027 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:12:01.416980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:01.404140 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:12:02.407976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:02.407855 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:12:02.408898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:02.408039 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:12:02.733462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:02.733231 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:03.765651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:03.765577 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:12:03.767161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:03.767106 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:12:03.770295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:03.770223 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:12:04.395113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:04.395051 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:04.482398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:04.482332 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:12:04.504111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:04.504042 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:12:05.097128 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.097074 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:12:05.754847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.754769 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj"] Oct 30 18:12:05.763312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.763231 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.781537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.781453 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj"] Oct 30 18:12:05.788469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.788227 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"kube-root-ca.crt\"" Oct 30 18:12:05.788469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.788312 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"default-dockercfg-vlhvr\"" Oct 30 18:12:05.788469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.788392 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"openshift-service-ca.crt\"" Oct 30 18:12:05.792726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.792663 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"premium-w4-4ef6ac-kserve-self-signed-certs\"" Oct 30 18:12:05.795021 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.794879 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:05.818440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.818335 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-home\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.818832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.818567 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-tmp-dir\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.818832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.818645 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ae5481-ba97-4cea-b38e-563d3b239908-tls-certs\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.818832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.818817 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5zdc\" (UniqueName: \"kubernetes.io/projected/61ae5481-ba97-4cea-b38e-563d3b239908-kube-api-access-f5zdc\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.819107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.818852 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-dshm\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.819238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.819201 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-model-cache\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.921193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.921119 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5zdc\" (UniqueName: \"kubernetes.io/projected/61ae5481-ba97-4cea-b38e-563d3b239908-kube-api-access-f5zdc\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.921193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.921198 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-dshm\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.921585 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.921319 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-model-cache\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.921585 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.921540 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-home\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.921683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.921615 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-tmp-dir\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.921733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.921687 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ae5481-ba97-4cea-b38e-563d3b239908-tls-certs\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.922022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.921983 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-model-cache\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.922187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.922061 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-home\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.922258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.922207 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-tmp-dir\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.925754 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.925679 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-dshm\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.926094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.926060 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ae5481-ba97-4cea-b38e-563d3b239908-tls-certs\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:05.934886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.934681 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:05.943682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:05.943599 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5zdc\" (UniqueName: \"kubernetes.io/projected/61ae5481-ba97-4cea-b38e-563d3b239908-kube-api-access-f5zdc\") pod \"premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:06.103764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:06.103577 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:06.461275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:06.461215 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj"] Oct 30 18:12:06.463714 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:12:06.463646 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61ae5481_ba97_4cea_b38e_563d3b239908.slice/crio-7b2e1ac7962ff946963b3a0ecf5e3829c2fe3287ec9e190a2890d32bdec0fdd5 WatchSource:0}: Error finding container 7b2e1ac7962ff946963b3a0ecf5e3829c2fe3287ec9e190a2890d32bdec0fdd5: Status 404 returned error can't find the container with id 7b2e1ac7962ff946963b3a0ecf5e3829c2fe3287ec9e190a2890d32bdec0fdd5 Oct 30 18:12:07.114047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:07.113915 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" event={"ID":"61ae5481-ba97-4cea-b38e-563d3b239908","Type":"ContainerStarted","Data":"8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4"} Oct 30 18:12:07.114803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:07.114072 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" event={"ID":"61ae5481-ba97-4cea-b38e-563d3b239908","Type":"ContainerStarted","Data":"7b2e1ac7962ff946963b3a0ecf5e3829c2fe3287ec9e190a2890d32bdec0fdd5"} Oct 30 18:12:07.161589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:07.161461 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podStartSLOduration=2.161425493 podStartE2EDuration="2.161425493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:07.15823233 +0000 UTC m=+621.546760448" watchObservedRunningTime="2026-10-30 18:12:07.161425493 +0000 UTC m=+621.549953609" Oct 30 18:12:07.615838 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:07.615751 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:08.104133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:08.104063 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:09.248241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:09.248180 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:10.872833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:10.872763 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:12.554995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:12.554843 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:14.271561 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:14.271461 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:16.113992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:16.113104 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:16.460223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:16.460126 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:16.463250 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:16.463173 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:16.543792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:16.535775 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:16.763175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:16.763012 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:16.900611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:16.900543 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:18.245227 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:18.245155 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:19.847018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:19.846751 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:19.974508 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:12:19.974379 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:12:21.555036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:21.551132 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:23.126478 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:23.126391 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:24.736026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:24.735890 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:26.342450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:26.342362 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:26.460414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:26.459450 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:12:28.038990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:28.035555 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:29.644980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:29.643536 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:31.322375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:31.322302 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:32.909092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:32.909003 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:34.488897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:34.488813 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:36.126284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:36.126211 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:37.839161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:37.839107 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:38.363645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.363567 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx"] Oct 30 18:12:38.370720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.370659 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.395785 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.395087 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx"] Oct 30 18:12:38.465622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.465532 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.465622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.465635 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxc74\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-kube-api-access-zxc74\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.466125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.465716 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.466125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.466024 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.466125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.466075 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee5582b-6f2b-41b6-82df-39fa47df1265-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.466243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.466147 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.466398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.466360 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.466588 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.466562 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.466660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.466642 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.568570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.568496 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.568570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.568576 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.568616 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.568711 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.568733 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxc74\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-kube-api-access-zxc74\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.568786 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.569021 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.569058 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee5582b-6f2b-41b6-82df-39fa47df1265-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.569121 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.569366 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569518 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.569492 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.569597 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.569772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.569740 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.570278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.570234 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee5582b-6f2b-41b6-82df-39fa47df1265-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.573101 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.573033 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.573350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.573264 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.592334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.592257 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.592334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.592320 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxc74\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-kube-api-access-zxc74\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.641202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.641127 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f"] Oct 30 18:12:38.648707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.648650 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.669059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.664441 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ce5eb-openshift-default-dockercfg-td7nf\"" Oct 30 18:12:38.669059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.667421 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f"] Oct 30 18:12:38.732582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.732515 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:38.774404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.774288 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.774874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.774557 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.775075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.775021 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzrn\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-kube-api-access-tpzrn\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.775152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.775133 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.775257 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.775231 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.775346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.775323 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.775446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.775423 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.775570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.775541 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.775632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.775592 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69b18c-b172-4310-a145-8385d76f1cbd-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.877342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.877260 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.877342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.877348 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69b18c-b172-4310-a145-8385d76f1cbd-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.877439 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.877520 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.877750 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzrn\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-kube-api-access-tpzrn\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.877814 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.877910 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.878015 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.878061 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.878108 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.878509 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.878908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.878877 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.879347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.879314 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.881469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.881274 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69b18c-b172-4310-a145-8385d76f1cbd-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.883497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.883438 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.885019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.884899 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.908449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.908246 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzrn\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-kube-api-access-tpzrn\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.909444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.909156 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:38.980704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:38.980627 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:39.119372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.118731 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx"] Oct 30 18:12:39.147325 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:12:39.147025 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee5582b_6f2b_41b6_82df_39fa47df1265.slice/crio-43d1480605ddc2a55a036850d095cf1599eb4696e9eaeede751ad84d89fcdd02 WatchSource:0}: Error finding container 43d1480605ddc2a55a036850d095cf1599eb4696e9eaeede751ad84d89fcdd02: Status 404 returned error can't find the container with id 43d1480605ddc2a55a036850d095cf1599eb4696e9eaeede751ad84d89fcdd02 Oct 30 18:12:39.152591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.152508 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:39.152884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.152661 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:39.152884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.152702 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:39.352261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.352179 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f"] Oct 30 18:12:39.356394 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:12:39.356328 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f69b18c_b172_4310_a145_8385d76f1cbd.slice/crio-082dc991ef36c5ffba284d5ffbf98740270e002c5e0e9d93c6343d48b72b2631 WatchSource:0}: Error finding container 082dc991ef36c5ffba284d5ffbf98740270e002c5e0e9d93c6343d48b72b2631: Status 404 returned error can't find the container with id 082dc991ef36c5ffba284d5ffbf98740270e002c5e0e9d93c6343d48b72b2631 Oct 30 18:12:39.362397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.362298 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:39.362811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.362488 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:39.362811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.362551 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:12:39.421977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.421659 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:39.654191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.654108 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" event={"ID":"0ee5582b-6f2b-41b6-82df-39fa47df1265","Type":"ContainerStarted","Data":"4fc181797cc9cfd2850f57390c80ec7c99d91a064bedd6156b7efcdd77ada93f"} Oct 30 18:12:39.654570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.654251 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" event={"ID":"0ee5582b-6f2b-41b6-82df-39fa47df1265","Type":"ContainerStarted","Data":"43d1480605ddc2a55a036850d095cf1599eb4696e9eaeede751ad84d89fcdd02"} Oct 30 18:12:39.659123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.658910 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" event={"ID":"9f69b18c-b172-4310-a145-8385d76f1cbd","Type":"ContainerStarted","Data":"1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8"} Oct 30 18:12:39.659449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.659173 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" event={"ID":"9f69b18c-b172-4310-a145-8385d76f1cbd","Type":"ContainerStarted","Data":"082dc991ef36c5ffba284d5ffbf98740270e002c5e0e9d93c6343d48b72b2631"} Oct 30 18:12:39.700123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.699744 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podStartSLOduration=1.699704589 podStartE2EDuration="1.699704589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:39.695598346 +0000 UTC m=+654.084126463" watchObservedRunningTime="2026-10-30 18:12:39.699704589 +0000 UTC m=+654.088232705" Oct 30 18:12:39.733527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.733455 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:39.741878 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.741752 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podStartSLOduration=1.74170918 podStartE2EDuration="1.74170918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:39.740772535 +0000 UTC m=+654.129300653" watchObservedRunningTime="2026-10-30 18:12:39.74170918 +0000 UTC m=+654.130237298" Oct 30 18:12:39.792582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.792476 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:39.981794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:39.981572 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:40.756707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:40.756617 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:40.757363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:40.757279 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:40.938988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:40.930743 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:41.011107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:41.008140 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:41.011107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:41.008242 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:41.749986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:41.740914 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:41.749986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:41.741088 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:41.982906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:41.982815 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:41.983549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:41.983488 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:42.576988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:42.546145 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:12:42.740290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:42.740206 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:42.740836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:42.740769 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:42.775265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:42.775148 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:42.984988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:42.983873 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:42.984988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:42.984065 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:43.067299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.065918 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:43.108985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.108767 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:43.165552 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.165317 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:43.239109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.232220 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:43.561119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.550398 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:12:43.561119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.552142 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:12:43.925988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.886164 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:43.925988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.886289 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:43.967987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:43.965748 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:12:44.183264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.173188 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:44.183264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.173313 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:44.248016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.219196 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:44.248016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.219341 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:44.248016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.246778 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:12:44.434133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.431178 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:44.434133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.432074 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:12:44.565562 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.565464 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:44.933114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.848496 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:44.933114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:44.848618 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:45.083144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.075252 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:12:45.083144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.075854 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:45.130326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.130192 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:45.130771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.130375 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:45.148222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.148125 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:12:45.150826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.150740 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:12:45.176260 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.176170 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:12:45.236100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.233702 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:12:45.247233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.247148 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:45.480133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.480037 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:12:45.534638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.534427 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:45.755032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.752774 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:45.755032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:45.753005 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:46.008221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:46.008135 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:46.013205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:46.013119 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:46.453995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:46.448340 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:46.750405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:46.750189 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:46.750907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:46.750841 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:46.981410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:46.981340 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:46.982117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:46.982015 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:47.755981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:47.747864 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:47.755981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:47.748043 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:47.982124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:47.982022 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:47.982506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:47.982149 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:48.140985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:48.137574 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:48.761215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:48.761131 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:48.762633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:48.762537 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:48.981628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:48.981559 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:48.982014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:48.981667 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:49.738136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:49.738035 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:49.742979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:49.742875 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:12:49.743303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:49.743034 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:12:49.989980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:49.989745 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 18:12:50.038236 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:50.038133 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 18:12:50.200528 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:12:50.200416 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:12:51.298724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:51.298656 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:51.301281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:51.301187 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:51.308617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:51.308463 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:12:51.339242 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:51.339067 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:51.779642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:51.779248 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:12:51.779642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:51.779385 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:12:52.406927 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:52.406848 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:52.415643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:52.413703 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:52.418326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:52.418190 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:12:52.941988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:52.937009 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:54.526089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:54.525912 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:56.140243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:56.140166 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:57.650811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:57.650751 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:59.332285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:12:59.332208 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:00.158484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:00.158401 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:00.248495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:00.248415 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:00.942322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:00.942244 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:02.497161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:02.497095 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:04.023537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:04.023443 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:05.567432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:05.567350 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:07.061604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:07.061511 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:08.637699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.637617 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:08.724154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.724067 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:13:08.778249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.778167 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk"] Oct 30 18:13:08.784479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.784422 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.806866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.805705 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef13cc-openshift-default-dockercfg-5k66q\"" Oct 30 18:13:08.822751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.822683 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk"] Oct 30 18:13:08.898511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.898337 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.899144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.899087 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.899480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.899207 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.899665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.899498 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.899665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.899635 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.899875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.899828 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.899999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.899921 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.900079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.900024 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92t7\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-kube-api-access-j92t7\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:08.900180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:08.900159 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.002571 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.002478 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.002955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.002618 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.002955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.002682 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.002955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.002781 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.002955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.002873 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.003202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.002957 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.003202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.003011 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.003202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.003044 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92t7\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-kube-api-access-j92t7\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.003202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.003166 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.003396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.003362 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.003447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.003394 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.003501 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.003437 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.003847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.003798 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.004489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.004452 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.007231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.007150 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.007558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.007481 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.026425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.026339 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.026849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.026359 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92t7\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-kube-api-access-j92t7\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.114035 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.113955 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:09.249517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.241285 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p"] Oct 30 18:13:09.250391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.250142 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.284577 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.276492 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p"] Oct 30 18:13:09.411600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.411518 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4740229c-ee94-4644-8678-66ab753340dd-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.412010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.411634 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.412010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.411773 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4740229c-ee94-4644-8678-66ab753340dd-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.412010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.411857 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.412010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.411919 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfkk\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-kube-api-access-8xfkk\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.412254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.412132 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.412489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.412372 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.412606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.412501 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.412698 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.412679 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.489801 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.489725 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk"] Oct 30 18:13:09.491958 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:13:09.491877 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0177a2_63ed_4c8a_887a_100f3e9c02c3.slice/crio-714d919a10cd16812fcace84e67a07c921e8e8853fdb342eaee078dd36b61e23 WatchSource:0}: Error finding container 714d919a10cd16812fcace84e67a07c921e8e8853fdb342eaee078dd36b61e23: Status 404 returned error can't find the container with id 714d919a10cd16812fcace84e67a07c921e8e8853fdb342eaee078dd36b61e23 Oct 30 18:13:09.496674 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.496588 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:09.497091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.496752 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:09.497091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.496806 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:09.514696 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.514617 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4740229c-ee94-4644-8678-66ab753340dd-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.514733 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.514766 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4740229c-ee94-4644-8678-66ab753340dd-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.514804 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.514845 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfkk\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-kube-api-access-8xfkk\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.514921 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.515060 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.515143 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.515294 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.515551 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.515651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.515606 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.516029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.515885 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.516320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.516282 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.516627 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.516598 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4740229c-ee94-4644-8678-66ab753340dd-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.521065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.520975 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.521413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.521310 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4740229c-ee94-4644-8678-66ab753340dd-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.538710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.538627 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.539132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.539044 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfkk\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-kube-api-access-8xfkk\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:09.632510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:09.632445 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:10.492615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:10.490220 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" event={"ID":"3b0177a2-63ed-4c8a-887a-100f3e9c02c3","Type":"ContainerStarted","Data":"8232a07492cde1cfa2442c3cec525f11c4c0b8dc528a7ef3ef2ea9fa7a33bb35"} Oct 30 18:13:10.492615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:10.490344 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" event={"ID":"3b0177a2-63ed-4c8a-887a-100f3e9c02c3","Type":"ContainerStarted","Data":"714d919a10cd16812fcace84e67a07c921e8e8853fdb342eaee078dd36b61e23"} Oct 30 18:13:10.566026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:10.562443 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podStartSLOduration=2.5623920289999997 podStartE2EDuration="2.562392029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:10.556156303 +0000 UTC m=+684.944684421" watchObservedRunningTime="2026-10-30 18:13:10.562392029 +0000 UTC m=+684.950920148" Oct 30 18:13:10.607779 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:10.607413 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:10.756348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:10.738032 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p"] Oct 30 18:13:10.786704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:10.786589 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:10.815027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:10.797682 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:10.815027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:10.797820 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:11.123745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:11.123672 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:11.137317 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:11.137007 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:11.161008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:11.141094 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:11.625604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:11.625502 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" event={"ID":"4740229c-ee94-4644-8678-66ab753340dd","Type":"ContainerStarted","Data":"786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f"} Oct 30 18:13:11.660309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:11.660229 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" event={"ID":"4740229c-ee94-4644-8678-66ab753340dd","Type":"ContainerStarted","Data":"9ea0b3f31acb38c7891e28f8eaf8259e2db56fa2dc30f9a40e02bed447085ecd"} Oct 30 18:13:11.660969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:11.660886 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:11.708063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:11.707869 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podStartSLOduration=2.7078308829999997 podStartE2EDuration="2.707830883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:11.6931279 +0000 UTC m=+686.081656018" watchObservedRunningTime="2026-10-30 18:13:11.707830883 +0000 UTC m=+686.096359001" Oct 30 18:13:12.128978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:12.124819 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:12.128978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:12.124987 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:12.226913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:12.226837 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:12.657014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:12.649052 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:12.657014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:12.649161 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:13.139330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:13.139237 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:13.139780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:13.139378 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:13.636017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:13.635878 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:13.646065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:13.645973 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:13.874790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:13.874686 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:14.122135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:14.119556 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:14.122135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:14.119685 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:14.645992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:14.640634 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:14.645992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:14.640773 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:15.152994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:15.139211 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:15.152994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:15.139341 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:15.563761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:15.563568 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:15.642974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:15.642856 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:15.645222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:15.645137 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:16.145016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:16.144255 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:16.145016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:16.144389 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:16.666121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:16.658358 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:16.666121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:16.658526 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:17.159912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:17.159818 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:17.160447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:17.160386 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:17.255062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:17.245443 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:17.647996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:17.647832 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:17.648402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:17.648017 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:18.248887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:18.248795 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:18.262224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:18.262107 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:18.645974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:18.644790 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:18.645974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:18.644978 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:18.864967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:18.861879 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:19.139077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:19.131489 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:13:19.139077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:19.131610 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:13:19.646999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:19.646339 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:19.647987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:19.646482 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:20.092836 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:13:20.092739 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:13:20.426870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:20.424974 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:20.430254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:20.430156 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:20.439921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:20.439858 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:13:20.442549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:20.442349 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:20.660044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:20.656120 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 18:13:20.660044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:20.656230 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 18:13:21.643926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:21.643826 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:21.645553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:21.645486 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:21.647486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:21.647421 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:13:22.134859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:22.134580 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:23.736573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:23.736512 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:25.312731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:25.312659 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:26.924659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:26.924580 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:27.927105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:27.927010 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv"] Oct 30 18:13:27.933021 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:27.932974 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:27.951672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:27.951606 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv"] Oct 30 18:13:27.957316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:27.957231 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:27.958048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:27.957881 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:13:28.014060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.013959 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsh8\" (UniqueName: \"kubernetes.io/projected/f1509c67-2df2-4bca-bf50-53e16c3c7059-kube-api-access-wvsh8\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.014452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.014188 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.014542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.014458 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.116485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.116410 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.116886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.116580 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.116886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.116827 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsh8\" (UniqueName: \"kubernetes.io/projected/f1509c67-2df2-4bca-bf50-53e16c3c7059-kube-api-access-wvsh8\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.117867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.117815 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.121526 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.121433 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-xks-ca\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.139775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.139713 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsh8\" (UniqueName: \"kubernetes.io/projected/f1509c67-2df2-4bca-bf50-53e16c3c7059-kube-api-access-wvsh8\") pod \"maas-api-revoke-sub-352e465b-4vfjv\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.264165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.263966 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:28.540127 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.539923 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:28.678259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:28.671595 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv"] Oct 30 18:13:28.678259 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:13:28.677422 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1509c67_2df2_4bca_bf50_53e16c3c7059.slice/crio-d7d4dcaa668baaa79bdada6498328e136ecde2b24fcb4a7c46a1eb66a5e84050 WatchSource:0}: Error finding container d7d4dcaa668baaa79bdada6498328e136ecde2b24fcb4a7c46a1eb66a5e84050: Status 404 returned error can't find the container with id d7d4dcaa668baaa79bdada6498328e136ecde2b24fcb4a7c46a1eb66a5e84050 Oct 30 18:13:29.493622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:29.493545 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" event={"ID":"f1509c67-2df2-4bca-bf50-53e16c3c7059","Type":"ContainerStarted","Data":"d7d4dcaa668baaa79bdada6498328e136ecde2b24fcb4a7c46a1eb66a5e84050"} Oct 30 18:13:30.228223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:30.228138 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:31.837544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:31.837466 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:31.950599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:31.950474 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:32.038295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:32.038204 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:32.590483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:32.590285 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" event={"ID":"f1509c67-2df2-4bca-bf50-53e16c3c7059","Type":"ContainerStarted","Data":"ce7b518a44175b72d592dee87adf51c6619c16d63b85340ba846f22df2e17543"} Oct 30 18:13:32.636326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:32.636198 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" podStartSLOduration=2.721447414 podStartE2EDuration="5.636163744s" totalImagesPullingTime="2.91471633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:32.632254644 +0000 UTC m=+707.020782763" watchObservedRunningTime="2026-10-30 18:13:32.636163744 +0000 UTC m=+707.024691905" Oct 30 18:13:33.445610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:33.445534 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:33.615725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:33.615623 5624 generic.go:356] "Generic (PLEG): container finished" podID="f1509c67-2df2-4bca-bf50-53e16c3c7059" containerID="ce7b518a44175b72d592dee87adf51c6619c16d63b85340ba846f22df2e17543" exitCode=0 Oct 30 18:13:33.616325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:33.616281 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" event={"ID":"f1509c67-2df2-4bca-bf50-53e16c3c7059","Type":"ContainerDied","Data":"ce7b518a44175b72d592dee87adf51c6619c16d63b85340ba846f22df2e17543"} Oct 30 18:13:34.909718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:34.909548 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:34.996397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:34.996305 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-openshift-ca\") pod \"f1509c67-2df2-4bca-bf50-53e16c3c7059\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " Oct 30 18:13:34.996796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:34.996768 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-xks-ca\") pod \"f1509c67-2df2-4bca-bf50-53e16c3c7059\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " Oct 30 18:13:34.996965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:34.996904 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1509c67-2df2-4bca-bf50-53e16c3c7059-kube-api-access-wvsh8\" (UniqueName: \"kubernetes.io/projected/f1509c67-2df2-4bca-bf50-53e16c3c7059-kube-api-access-wvsh8\") pod \"f1509c67-2df2-4bca-bf50-53e16c3c7059\" (UID: \"f1509c67-2df2-4bca-bf50-53e16c3c7059\") " Oct 30 18:13:34.997133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:34.997084 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-openshift-ca" pod "f1509c67-2df2-4bca-bf50-53e16c3c7059" (UID: "f1509c67-2df2-4bca-bf50-53e16c3c7059"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:34.998344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:34.998287 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:13:35.002464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:35.002381 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1509c67-2df2-4bca-bf50-53e16c3c7059-kube-api-access-wvsh8" pod "f1509c67-2df2-4bca-bf50-53e16c3c7059" (UID: "f1509c67-2df2-4bca-bf50-53e16c3c7059"). InnerVolumeSpecName "kube-api-access-wvsh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:35.003223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:35.003171 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-xks-ca" pod "f1509c67-2df2-4bca-bf50-53e16c3c7059" (UID: "f1509c67-2df2-4bca-bf50-53e16c3c7059"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:35.050143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:35.049905 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:35.100136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:35.100063 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1509c67-2df2-4bca-bf50-53e16c3c7059-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:13:35.100136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:35.100125 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvsh8\" (UniqueName: \"kubernetes.io/projected/f1509c67-2df2-4bca-bf50-53e16c3c7059-kube-api-access-wvsh8\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:13:35.760300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:35.749613 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" event={"ID":"f1509c67-2df2-4bca-bf50-53e16c3c7059","Type":"ContainerDied","Data":"d7d4dcaa668baaa79bdada6498328e136ecde2b24fcb4a7c46a1eb66a5e84050"} Oct 30 18:13:35.776974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:35.751120 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv" Oct 30 18:13:35.818830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:35.818724 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d4dcaa668baaa79bdada6498328e136ecde2b24fcb4a7c46a1eb66a5e84050" Oct 30 18:13:37.269997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.269824 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.311991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.311782 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:13:37.315988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.312676 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Oct 30 18:13:37.315988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.314216 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:13:37.340153 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.340076 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:13:37.596907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.596668 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:37.651771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.651669 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:37.717426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.717353 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:37.718820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:37.718622 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.132983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.132161 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Oct 30 18:13:38.274016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.272433 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.276956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.275144 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:13:38.314626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.314514 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.316096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.315080 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:13:38.321846 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.321770 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:13:38.322551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.322516 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:13:38.377025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.368544 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.857996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.753297 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:38.909073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:38.860370 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:13:39.473448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.444762 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:13:39.542987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.522415 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:39.542987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.530003 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.746109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.530599 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:13:39.909977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.894064 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:39.999512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.587565 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.000187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.630570 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.019141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.553395 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.045302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.918553 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.065234 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:39.931233 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" podUID="f339d950-7963-41f3-99ea-237bca9464dd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.63:50052\" within 1s: context deadline exceeded" Oct 30 18:13:40.065704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.049404 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.62:50052\" within 1s: context deadline exceeded" Oct 30 18:13:40.275763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.275561 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:13:40.276332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.276264 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.327455 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:13:40.293999 5624 logging.go:55] [core] [Channel #299 SubChannel #302]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.63:50052", ServerName: "10.134.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.63:50052: operation was canceled" Oct 30 18:13:40.647131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.645812 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.736992 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:13:40.674591 5624 logging.go:55] [core] [Channel #300 SubChannel #301]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.62:50052", ServerName: "10.134.0.62:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:56164->10.134.0.62:50052: use of closed network connection" Oct 30 18:13:40.768266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.696111 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" podUID="0a09133a-00c6-4632-ba5e-b859f8da282e" containerName="server" probeResult="failure" output="Get \"https://10.134.0.37:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.772319 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.772247 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.859921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.742396 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:40.941431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.772132 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g"] Oct 30 18:13:40.941431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.938094 5624 cpu_manager.go:414] "containerMap: removing container" podUID="f1509c67-2df2-4bca-bf50-53e16c3c7059" containerName="revoke-keys" Oct 30 18:13:40.941431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.938143 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1509c67-2df2-4bca-bf50-53e16c3c7059" containerName="revoke-keys" Oct 30 18:13:40.941431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.938408 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1509c67-2df2-4bca-bf50-53e16c3c7059" containerName="revoke-keys" Oct 30 18:13:41.019039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.998883 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.490144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.698689 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:41.490144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.810595 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-769875db56-84jnk" podUID="dc1ecec4-614d-4cbf-978b-7b0fdb293d0f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.38:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.544240 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:41.526330 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:41.544240 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:40.864519 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.719984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:41.685068 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.757099 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:41.690803 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:41.786983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:41.785605 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.834226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:41.834125 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.844989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:41.844121 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:41.898563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:41.898502 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.002528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.002419 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:42.029116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.025451 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.044213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.044128 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.045155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.045115 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.077299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.073139 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.093610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.093171 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.251086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.250984 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.251650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.251604 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.252066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.252024 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17604251-a04e-4687-8bd9-c164880f103a-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.252291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.252255 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.252596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.252573 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.252768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.252748 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9j9\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-kube-api-access-pz9j9\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.253159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.253134 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.253502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.253457 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.270390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.270310 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17604251-a04e-4687-8bd9-c164880f103a-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.272920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.272557 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.284643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.274778 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.289095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.275260 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.289497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.275305 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.359307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.320459 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:13:42.360039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.359972 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:13:42.379199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.379116 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g"] Oct 30 18:13:42.380285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.380216 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:13:42.384170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.384121 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:13:42.384653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.383853 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17604251-a04e-4687-8bd9-c164880f103a-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.390063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.334605 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.390654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.390600 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9j9\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-kube-api-access-pz9j9\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.395043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.391196 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17604251-a04e-4687-8bd9-c164880f103a-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.419350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.396086 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.421049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.396443 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17604251-a04e-4687-8bd9-c164880f103a-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.561155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.547247 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9j9\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-kube-api-access-pz9j9\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:42.780982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.762775 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:43.014583 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.877561 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:43.014583 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:42.954295 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:13:43.014583 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:43.005626 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:13:43.044986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:43.034125 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:43.067705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:43.056228 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:13:43.252455 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:43.252355 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:13:43.387731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:43.299044 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:43.400915 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:43.389099 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:43.401473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:43.399196 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17604251-a04e-4687-8bd9-c164880f103a-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:43.966490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:43.966372 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.100412 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.100307 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.126294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.124910 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.131794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.100727 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.247467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.189435 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:44.527113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.526861 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:13:44.830100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.821229 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:44.959999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.959869 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:13:44.960990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.960892 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:13:44.961312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:44.961030 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:13:45.150013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.132118 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:13:45.150013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.132183 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:13:45.150013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.132118 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:13:45.203685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.203604 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:13:45.230522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.230453 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:13:45.299233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.298604 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:45.300385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.300335 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:13:45.457146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.457079 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g"] Oct 30 18:13:45.578887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.578790 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:45.579568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.579500 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:45.579858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:45.579832 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:13:46.028904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:46.024084 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" event={"ID":"17604251-a04e-4687-8bd9-c164880f103a","Type":"ContainerStarted","Data":"ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c"} Oct 30 18:13:46.028904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:46.024212 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" event={"ID":"17604251-a04e-4687-8bd9-c164880f103a","Type":"ContainerStarted","Data":"1f0e8c30d5622fbe240970dcc8d81242d64e45db042b48bacfc3f6998245eaf4"} Oct 30 18:13:46.096770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:46.096634 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podStartSLOduration=8.096599412 podStartE2EDuration="8.096599412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:46.09190009 +0000 UTC m=+720.480428191" watchObservedRunningTime="2026-10-30 18:13:46.096599412 +0000 UTC m=+720.485127529" Oct 30 18:13:46.480729 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:46.480651 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:46.484755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:46.484681 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:46.485167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:46.484822 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:13:47.060544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:47.060466 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:47.481210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:47.481139 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:47.482014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:47.481330 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:13:48.480357 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:48.480282 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:48.480749 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:48.480411 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:13:48.806230 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:48.806038 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:49.480626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.480448 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:49.480626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.480570 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:13:49.779448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.779260 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5"] Oct 30 18:13:49.823541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.823453 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:49.844541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.834072 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:49.844994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.844607 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"distinct-w5-fdf4da-kserve-self-signed-certs\"" Oct 30 18:13:49.859164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.856377 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"openshift-service-ca.crt\"" Oct 30 18:13:49.859164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.856566 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"default-dockercfg-5qwn9\"" Oct 30 18:13:49.863704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.862336 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5"] Oct 30 18:13:49.866918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.865118 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"kube-root-ca.crt\"" Oct 30 18:13:49.928640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.927509 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-home\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:49.928640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.927703 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-dshm\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:49.928640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.927759 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c9af44-a4f5-4014-aac6-9116897541e0-tls-certs\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:49.928640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.928056 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-tmp-dir\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:49.928640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.928142 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-model-cache\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:49.928640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:49.928299 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkv5\" (UniqueName: \"kubernetes.io/projected/16c9af44-a4f5-4014-aac6-9116897541e0-kube-api-access-llkv5\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.045116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.032798 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-tmp-dir\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.068985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.054838 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-tmp-dir\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.068985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.055022 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-model-cache\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.068985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.055226 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llkv5\" (UniqueName: \"kubernetes.io/projected/16c9af44-a4f5-4014-aac6-9116897541e0-kube-api-access-llkv5\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.068985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.055495 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-home\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.068985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.055617 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-dshm\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.068985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.055672 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c9af44-a4f5-4014-aac6-9116897541e0-tls-certs\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.068985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.058200 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-model-cache\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.079998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.072000 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-home\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.134980 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:13:50.132804 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:13:50.190199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.183878 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkv5\" (UniqueName: \"kubernetes.io/projected/16c9af44-a4f5-4014-aac6-9116897541e0-kube-api-access-llkv5\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.190199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.186900 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c9af44-a4f5-4014-aac6-9116897541e0-tls-certs\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.202998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.191332 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-dshm\") pod \"distinct-w5-fdf4da-kserve-8577fbc559-52sr5\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.256763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.256683 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:50.427290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.427206 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:50.539985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.519151 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:50.539985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.519266 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:13:50.877856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:50.877774 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5"] Oct 30 18:13:50.880448 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:13:50.880375 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16c9af44_a4f5_4014_aac6_9116897541e0.slice/crio-e938d2993f1fd992fc005dcfe76f929048532c3985e827cba55138214c80b423 WatchSource:0}: Error finding container e938d2993f1fd992fc005dcfe76f929048532c3985e827cba55138214c80b423: Status 404 returned error can't find the container with id e938d2993f1fd992fc005dcfe76f929048532c3985e827cba55138214c80b423 Oct 30 18:13:51.160014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:51.159303 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" event={"ID":"16c9af44-a4f5-4014-aac6-9116897541e0","Type":"ContainerStarted","Data":"5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09"} Oct 30 18:13:51.160014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:51.159387 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" event={"ID":"16c9af44-a4f5-4014-aac6-9116897541e0","Type":"ContainerStarted","Data":"e938d2993f1fd992fc005dcfe76f929048532c3985e827cba55138214c80b423"} Oct 30 18:13:51.219981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:51.219032 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podStartSLOduration=2.218985047 podStartE2EDuration="2.218985047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:51.217538762 +0000 UTC m=+725.606066882" watchObservedRunningTime="2026-10-30 18:13:51.218985047 +0000 UTC m=+725.607513164" Oct 30 18:13:51.256914 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:51.256840 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:13:51.481051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:51.480838 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:51.481051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:51.480948 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:13:52.022957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:52.022891 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:52.480379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:52.480297 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:52.480831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:52.480396 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:13:53.480665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:53.480467 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:13:53.480665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:53.480566 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 18:13:53.723751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:53.723646 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:54.600224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:54.600163 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:54.602977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:54.602856 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:54.607550 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:54.607463 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:13:55.321042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:55.320899 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:56.939095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:56.939025 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:58.637263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:58.637187 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:58.781519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:13:58.781428 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:14:00.263473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:00.263379 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:14:00.287127 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:00.287056 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:00.912856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:00.912764 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:14:01.206111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:01.190464 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:01.274424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:01.274306 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:01.481091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:01.477170 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:01.481091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:01.477335 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:01.846327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:01.846107 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:01.902994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:01.902142 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:01.914283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:01.914182 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:02.533112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.503974 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:02.533112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.507736 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:14:02.666620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.611897 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:02.666620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.613390 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:14:02.666620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.660338 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:02.903460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.847068 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:02.916985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.916881 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:14:02.936521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.923294 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:02.952150 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.923374 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:02.990986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:02.954773 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:14:03.360512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:03.360335 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:14:03.680181 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:03.680084 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:03.694178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:03.694087 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:14:03.973402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:03.948609 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:03.976478 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:03.976372 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:14:04.042978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.021251 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:04.135602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.135509 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:04.169426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.143003 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:04.222124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.222050 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:14:04.265392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.264568 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:14:04.356299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.356213 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:14:04.358565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.358480 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:14:04.366977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.365407 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:04.442972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.432807 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:14:04.488132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:04.485423 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:05.932353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:05.932280 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:06.192291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:06.192119 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:07.550017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:07.546853 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:07.934473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:07.934411 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:08.620698 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.620601 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc"] Oct 30 18:14:08.632519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.632420 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.692501 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.692419 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc"] Oct 30 18:14:08.785877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.785792 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.786406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.786360 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.786625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.786453 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.786720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.786666 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.786835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.786787 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7k5g\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-kube-api-access-c7k5g\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.786908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.786881 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.787039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.787011 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.787421 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.787368 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.787533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.787465 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.890110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.890023 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.890518 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.890160 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.890518 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.890497 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.890673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.890530 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.891732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.891007 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.891732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.891187 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.891732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.891313 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7k5g\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-kube-api-access-c7k5g\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.891732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.891456 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.891732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.891502 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.892320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.891859 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.892320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.891894 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.892320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.892295 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.892776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.892734 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.893057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.892791 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.896222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.896136 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.896699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.896648 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.916182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.916039 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:08.920017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:08.919812 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7k5g\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-kube-api-access-c7k5g\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:09.020819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:09.020704 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:09.357752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:09.357382 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:09.477856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:09.477400 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc"] Oct 30 18:14:09.503375 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:14:09.503294 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f10c059_42a2_4ac2_b8bf_0a7aee7061ce.slice/crio-8a1cf8deb021f211ba3499ebbca7a2b7202f07ee17e2b69951622f1608384fa0 WatchSource:0}: Error finding container 8a1cf8deb021f211ba3499ebbca7a2b7202f07ee17e2b69951622f1608384fa0: Status 404 returned error can't find the container with id 8a1cf8deb021f211ba3499ebbca7a2b7202f07ee17e2b69951622f1608384fa0 Oct 30 18:14:09.524687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:09.523408 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:09.524687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:09.523593 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:09.524687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:09.523644 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:09.603904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:09.603805 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" event={"ID":"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce","Type":"ContainerStarted","Data":"8a1cf8deb021f211ba3499ebbca7a2b7202f07ee17e2b69951622f1608384fa0"} Oct 30 18:14:10.633041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:10.632929 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" event={"ID":"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce","Type":"ContainerStarted","Data":"fe08d835d87bc1ba652ba539b73b240101d4a47824f5a86a29e9b4368124f595"} Oct 30 18:14:10.712988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:10.707310 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podStartSLOduration=2.707270008 podStartE2EDuration="2.707270008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:10.704784658 +0000 UTC m=+745.093313026" watchObservedRunningTime="2026-10-30 18:14:10.707270008 +0000 UTC m=+745.095798132" Oct 30 18:14:11.021613 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:11.021435 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:11.025202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:11.025139 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:11.025584 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:11.025242 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:14:11.228465 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:11.228400 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:12.022546 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:12.022472 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:12.023388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:12.022597 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:14:12.992989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:12.991665 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:13.021913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:13.021842 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:13.022263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:13.021977 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:14:14.021924 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:14.021841 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:14.022845 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:14.022032 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:14:14.632165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:14.632068 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:14.975578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:14.975414 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:15.028220 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:15.028144 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:15.029115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:15.028510 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:14:16.022214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:16.022132 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:16.022638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:16.022318 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:14:16.677683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:16.677610 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:17.022663 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:17.022462 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:14:17.022663 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:17.022567 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:14:18.335111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:18.335022 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:18.703744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:18.703669 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:18.704868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:18.704810 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:18.706955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:18.706870 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:14:19.934468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:19.934391 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:19.942360 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:14:19.942284 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:14:21.429058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:21.428927 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:22.915310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:22.915239 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:23.710216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.710135 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9"] Oct 30 18:14:23.719352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.719274 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.743595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.742084 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9"] Oct 30 18:14:23.748343 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.748277 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fde2c5-openshift-default-dockercfg-ffxp6\"" Oct 30 18:14:23.817704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.817599 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d"] Oct 30 18:14:23.830600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.830510 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2"] Oct 30 18:14:23.839553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.839480 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.847395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.847322 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d"] Oct 30 18:14:23.848085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.847914 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr"] Oct 30 18:14:23.848576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.848527 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.849739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.849662 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.850132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.849838 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7ps\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-kube-api-access-kv7ps\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.850132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.849891 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.850132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.849968 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.850346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.850308 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.850543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.850510 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.850753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.850726 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.850835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.850809 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.851066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.850922 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.865131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.865079 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2"] Oct 30 18:14:23.865865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.865808 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:23.870136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.870083 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr"] Oct 30 18:14:23.953576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953492 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.953576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953584 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a21199-1155-43cc-b36b-6215e6d3130b-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953696 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953732 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953767 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953797 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953837 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8db066-80e2-4003-919a-f92347b92fc4-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953859 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8db066-80e2-4003-919a-f92347b92fc4-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.953894 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gjm\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-kube-api-access-l8gjm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954025 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954142 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954233 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954320 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7ps\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-kube-api-access-kv7ps\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954355 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954389 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954459 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtg5h\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-kube-api-access-vtg5h\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954512 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954549 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954590 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954618 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954678 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954701 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954744 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954817 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.954815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954838 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a21199-1155-43cc-b36b-6215e6d3130b-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:23.956322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954881 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:23.956322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.954948 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.956322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.955957 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.956322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.956037 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.956322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.956303 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.956716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.956604 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.957808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.957754 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.962807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.962635 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.968485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.968337 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.980196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.980137 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:23.980576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:23.980429 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7ps\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-kube-api-access-kv7ps\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:24.054480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.054389 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:24.057849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.057772 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.058249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.057901 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.058249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.057958 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.058249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.057994 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.058249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058058 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.058249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058094 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a21199-1155-43cc-b36b-6215e6d3130b-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.058553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058267 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.058553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058373 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.058553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058497 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8db066-80e2-4003-919a-f92347b92fc4-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.058707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058556 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.058707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058597 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8db066-80e2-4003-919a-f92347b92fc4-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.058707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058694 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.058847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058743 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.058847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058792 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.059043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.058892 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.059128 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.059026 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.059128 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.059038 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.059210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.059176 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.059424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.059388 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a21199-1155-43cc-b36b-6215e6d3130b-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.059605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.059582 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtg5h\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-kube-api-access-vtg5h\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.059661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.059645 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.059807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.059769 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8db066-80e2-4003-919a-f92347b92fc4-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.060162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.060015 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.060253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.060164 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.060253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.060233 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.060372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.060324 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.060372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.060356 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.061214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.060695 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.061214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.060827 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.061214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.061092 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7sg\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-kube-api-access-fl7sg\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.061214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.061165 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.061214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.061209 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gjm\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-kube-api-access-l8gjm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.062972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.061351 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.062972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.061401 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a21199-1155-43cc-b36b-6215e6d3130b-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.062972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.061577 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.062972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.062121 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.062972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.062196 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.067799 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.067727 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.068781 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.068716 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.069139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.068717 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8db066-80e2-4003-919a-f92347b92fc4-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.069230 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.069202 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a21199-1155-43cc-b36b-6215e6d3130b-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.081632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.080767 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.081632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.080787 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.083856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.083790 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtg5h\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-kube-api-access-vtg5h\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.085605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.085552 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gjm\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-kube-api-access-l8gjm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.164881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.164749 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.164881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.164868 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.165302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.164954 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.165302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.165001 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.165302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.165064 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.165302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.165115 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7sg\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-kube-api-access-fl7sg\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.165536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.165511 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.166966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.166381 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.166966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.166481 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.166966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.166711 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.167475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.167056 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.167475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.167161 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.169557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.169239 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.169557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.169325 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.175045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.174986 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.177886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.177831 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.222326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.218640 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:24.227420 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.226403 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.232143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.229907 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7sg\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-kube-api-access-fl7sg\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.232143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.231587 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:24.246967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.246877 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:24.461528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.461463 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9"] Oct 30 18:14:24.539867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.539729 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.540229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.540005 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.540229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.540081 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.549340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.549265 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:24.699065 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:14:24.696571 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a21199_1155_43cc_b36b_6215e6d3130b.slice/crio-199af96dd577981d9a75b0e16be8620fdefb0230ef7dddc814074ee8ae15013b WatchSource:0}: Error finding container 199af96dd577981d9a75b0e16be8620fdefb0230ef7dddc814074ee8ae15013b: Status 404 returned error can't find the container with id 199af96dd577981d9a75b0e16be8620fdefb0230ef7dddc814074ee8ae15013b Oct 30 18:14:24.705012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.704914 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d"] Oct 30 18:14:24.718854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.718730 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.737272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.735991 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.737272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.736178 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.902727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.902655 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2"] Oct 30 18:14:24.904577 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:14:24.904515 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8db066_80e2_4003_919a_f92347b92fc4.slice/crio-b1830a53c2b8014de37faf6ae16863bf4ad1e73da09ae201756454cd80022a3c WatchSource:0}: Error finding container b1830a53c2b8014de37faf6ae16863bf4ad1e73da09ae201756454cd80022a3c: Status 404 returned error can't find the container with id b1830a53c2b8014de37faf6ae16863bf4ad1e73da09ae201756454cd80022a3c Oct 30 18:14:24.920503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.919398 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.920503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.919603 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.920503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.919679 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:24.926776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.926544 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" event={"ID":"30a21199-1155-43cc-b36b-6215e6d3130b","Type":"ContainerStarted","Data":"199af96dd577981d9a75b0e16be8620fdefb0230ef7dddc814074ee8ae15013b"} Oct 30 18:14:24.966830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.966725 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" event={"ID":"cd73c927-afa4-4023-bf6b-a04af2ea3a79","Type":"ContainerStarted","Data":"24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129"} Oct 30 18:14:24.967834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.966911 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" event={"ID":"cd73c927-afa4-4023-bf6b-a04af2ea3a79","Type":"ContainerStarted","Data":"18680fb00f6cb7709256c14b4ab6f77ea90805a37859ea5510f80256b0a20099"} Oct 30 18:14:24.987076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:24.986904 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr"] Oct 30 18:14:24.994640 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:14:24.994475 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97247502_e6a3_44b3_aef5_cd9a3c71dd64.slice/crio-b608d2f8ce0e304bf0d90764c8d2d924c55e7ef2f70598c004b6b26e99e41b67 WatchSource:0}: Error finding container b608d2f8ce0e304bf0d90764c8d2d924c55e7ef2f70598c004b6b26e99e41b67: Status 404 returned error can't find the container with id b608d2f8ce0e304bf0d90764c8d2d924c55e7ef2f70598c004b6b26e99e41b67 Oct 30 18:14:25.015595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:25.010080 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:25.015595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:25.010251 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:25.015595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:25.010320 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:25.058498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:25.055001 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:25.061764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:25.061686 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:25.062335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:25.062262 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:26.061074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.060060 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.061074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.060219 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:26.067539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.067469 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" event={"ID":"97247502-e6a3-44b3-aef5-cd9a3c71dd64","Type":"ContainerStarted","Data":"255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c"} Oct 30 18:14:26.068043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.067576 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" event={"ID":"97247502-e6a3-44b3-aef5-cd9a3c71dd64","Type":"ContainerStarted","Data":"b608d2f8ce0e304bf0d90764c8d2d924c55e7ef2f70598c004b6b26e99e41b67"} Oct 30 18:14:26.071924 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.071829 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" event={"ID":"3f8db066-80e2-4003-919a-f92347b92fc4","Type":"ContainerStarted","Data":"774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b"} Oct 30 18:14:26.072330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.071995 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" event={"ID":"3f8db066-80e2-4003-919a-f92347b92fc4","Type":"ContainerStarted","Data":"b1830a53c2b8014de37faf6ae16863bf4ad1e73da09ae201756454cd80022a3c"} Oct 30 18:14:26.079125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.079044 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" event={"ID":"30a21199-1155-43cc-b36b-6215e6d3130b","Type":"ContainerStarted","Data":"29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada"} Oct 30 18:14:26.179173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.179074 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podStartSLOduration=3.179041555 podStartE2EDuration="3.179041555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:26.173383751 +0000 UTC m=+760.561911868" watchObservedRunningTime="2026-10-30 18:14:26.179041555 +0000 UTC m=+760.567569671" Oct 30 18:14:26.180718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.180643 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podStartSLOduration=3.180615552 podStartE2EDuration="3.180615552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:25.026729556 +0000 UTC m=+759.415257676" watchObservedRunningTime="2026-10-30 18:14:26.180615552 +0000 UTC m=+760.569143668" Oct 30 18:14:26.233572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.232878 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:26.233572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.233250 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:26.249106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.249046 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:26.259071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.257369 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.259071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.257529 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:26.259071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.258311 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.259071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.258383 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:26.259071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.258533 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.259071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.258559 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:26.265498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.265435 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:26.276902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.276785 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podStartSLOduration=3.2767430060000002 podStartE2EDuration="3.276743006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:26.274689162 +0000 UTC m=+760.663217279" watchObservedRunningTime="2026-10-30 18:14:26.276743006 +0000 UTC m=+760.665271117" Oct 30 18:14:26.371120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:26.362107 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podStartSLOduration=3.362071877 podStartE2EDuration="3.362071877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:26.360852447 +0000 UTC m=+760.749380585" watchObservedRunningTime="2026-10-30 18:14:26.362071877 +0000 UTC m=+760.750599994" Oct 30 18:14:27.128993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.103917 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.302988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.212123 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:27.302988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.225600 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.302988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.225748 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:27.302988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.290903 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.302988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.291084 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:27.329027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.311454 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.329027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.311573 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:27.918980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:27.918809 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:28.062049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.061560 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.062049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.061674 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:28.075978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.070988 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:28.305113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.304751 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.305113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.304903 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:28.345990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.316153 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 18:14:28.371989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.359125 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:28.371989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.368569 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:14:28.371989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.369613 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.371989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.369716 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:28.372522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.371990 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:28.431984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.431028 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:28.456991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.444916 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:14:28.568097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.510112 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.568097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.510236 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:28.973432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.943360 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:28.986979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:28.978520 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:14:29.099310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.099201 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.158974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.157637 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:29.282152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.178547 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 18:14:29.282152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.178790 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:29.333084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.286867 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.333084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.287014 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:29.333084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.302232 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.333084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.302379 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:29.333084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.305529 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.333084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.305680 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:29.376008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.368870 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:14:29.410025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.381513 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:14:29.499264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.499119 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:29.656725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.505785 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:14:29.656725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.649415 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:14:29.656725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.649577 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:14:29.974103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:29.963276 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:14:30.188137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.188060 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" podUID="0a09133a-00c6-4632-ba5e-b859f8da282e" containerName="server" probeResult="failure" output="Get \"https://10.134.0.37:8443/healthz\": context deadline exceeded" Oct 30 18:14:30.273112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.196571 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.478346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.306781 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.479718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.479629 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:30.480047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.377842 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.568121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.378828 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.605820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.320255 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:30.606693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.531568 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.607204 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.607146 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:30.754997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.733388 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.764401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.764305 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:30.863163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.802633 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.863163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.847169 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:30.863163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:30.764412 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:31.825087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.780828 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:31.825087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.781018 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:31.851972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.851812 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:32.015729 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.956869 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:14:32.079114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.972354 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:14:32.079114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.972609 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 30 18:14:32.079114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.972735 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:14:32.110068 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.972799 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:32.111485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.972836 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:14:32.160456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:31.991746 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:14:32.173056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.015541 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.173405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.015646 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:32.173405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.173347 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.386008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.379191 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.572534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.572423 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:32.789117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.785644 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:32.793321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.793203 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:32.849108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.834640 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:32.849108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.834751 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:32.932113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.895067 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:32.932113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.895166 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:32.932113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.895757 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:32.932113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.895800 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:32.933194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:32.933047 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:33.199024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.149914 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:14:33.679986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.162465 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:33.679986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.184083 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:14:33.681358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.197831 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:33.704607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.200666 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:14:33.774358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.200743 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:33.778240 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.200769 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:33.843074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.260146 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:33.843074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.826701 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:33.843074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.311648 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:33.918669 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.266979 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:33.924585 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.263125 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:33.961099 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.960823 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:33.968168 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:14:33.488317 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.578s" Oct 30 18:14:33.980332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.972354 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:14:33.981133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.981090 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:14:34.024248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.438671 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:34.074880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.074781 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:34.077720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:33.773347 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:34.078471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.078402 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:34.082152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.046903 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:34.089235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.089137 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:34.089686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.084239 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.64:50052\" within 1s: context deadline exceeded" Oct 30 18:14:34.124050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.098782 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:34.124050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.098891 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:34.230130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.204759 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:34.269110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.267167 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:34.269110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.267290 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:34.269110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.267788 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:34.269110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.267843 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:34.302102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.302011 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:34.302715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:34.302644 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:34.411982 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:14:34.392202 5624 logging.go:55] [core] [Channel #379 SubChannel #380]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.64:50052", ServerName: "10.134.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: write tcp 10.134.0.2:55556->10.134.0.64:50052: use of closed network connection Oct 30 18:14:35.099836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.099735 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:14:35.132732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.103157 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:14:35.134423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.103224 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:14:35.140109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.103254 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:14:35.197056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.104840 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:35.230339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.104903 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:14:35.369104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.336668 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:35.369104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.336796 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:35.409341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.409228 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:35.523898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.506723 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:35.523898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.504126 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:35.549136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.536867 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:35.549804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.549709 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:35.633791 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.614723 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:35.633791 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.614847 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:35.718970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.645350 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:35.734511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.734423 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:35.735385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.653551 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:35.735699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.735657 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:35.759576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.759481 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:35.782999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:35.764805 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:36.086221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.066264 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:36.086221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.066384 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:36.145765 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.145651 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:36.147621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.147533 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:14:36.153983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.149071 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:14:36.153983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.149663 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:36.202988 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:14:36.201220 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.22s" Oct 30 18:14:36.219990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.218176 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:36.225202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.224441 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:36.225202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.224577 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:36.251024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.246818 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:36.251024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.246987 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:36.251024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.248032 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:36.251024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.248123 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:36.251024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.248234 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:14:36.726024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.725432 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:14:36.726024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.725564 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:14:36.726024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.725678 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:14:36.726024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.725734 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:14:36.795097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.787161 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:36.795097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:36.787304 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:37.093092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.089205 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:37.093672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.093583 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:37.126659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.125176 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:37.195736 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.195634 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:14:37.271251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.271111 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:37.301271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.301153 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:37.325283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.325153 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:37.351682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.329697 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:37.461909 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.458077 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:14:37.545163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.540299 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:37.545163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.540418 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:37.634224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.623332 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:37.634224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.623459 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:37.634224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.623599 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:37.634224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.623625 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:37.634224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.623700 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:37.634224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.623739 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:37.670871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:37.670781 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:38.230875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.230762 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:14:38.294290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.294179 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:38.295201 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.295121 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:38.362494 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.362381 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:38.367032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.366913 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:38.516111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.440015 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:38.517417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.416741 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:38.518624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.440722 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:38.519022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.416766 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:38.519236 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.519189 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:38.519380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.469603 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Oct 30 18:14:38.519560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.379208 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:14:38.531982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.523869 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:38.535328 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.531679 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:38.645137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.624140 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:38.654093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:38.650807 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:38.882109 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:14:38.855606 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.062s" Oct 30 18:14:39.304568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.304363 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:39.376573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.376435 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 18:14:39.403345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.403250 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.405039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.404914 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:39.522001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.497075 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:39.579108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.575567 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:39.599982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.574132 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:14:39.601767 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.504371 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:39.602448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.602364 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:39.602747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.591242 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:39.603096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.522646 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.603814 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.504535 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:39.604092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.604039 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:39.665517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.662335 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:39.665517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.663323 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:14:39.925091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:39.924979 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:40.220109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.139892 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:40.220109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.212564 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:40.305341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.305255 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:40.306682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.306569 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:40.326002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.325826 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:40.326554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.326479 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:40.405763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.352824 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:40.406535 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.405353 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.449997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.407052 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:14:40.558622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.558404 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:40.559147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.559073 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:40.654459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.632771 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:40.654459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.633094 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:14:40.654459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.633850 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:40.698992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.689699 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:40.772854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.772756 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:40.777190 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.707576 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:14:40.785987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:40.707481 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.103981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.102812 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:41.103981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.103007 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:41.237300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.237230 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:41.237887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.237814 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:41.238177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.238156 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:41.238295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.238275 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:41.268995 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:14:41.246049 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.031s" Oct 30 18:14:41.268995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.246233 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:41.268995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.246275 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:41.268995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.246359 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:14:41.293992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.287804 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:41.293992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.287979 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:41.523887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.523772 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:14:41.949828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.751284 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.951314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.774924 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:14:41.952990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.775030 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:41.987045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.878545 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 18:14:42.755209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:41.902197 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:42.844319 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:42.042718 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.109825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:42.239129 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Oct 30 18:14:43.109825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:42.294256 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.110278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.109840 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:43.110278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:42.935513 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:43.110278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:42.378369 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.110636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:42.294357 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.110885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.110829 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:43.114102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.045174 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.116451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.084713 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.121538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:42.294417 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.122293 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.122211 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:43.122580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:42.298858 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.122714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.122687 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:43.122824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.084801 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:14:43.128194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.084845 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.131770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.124661 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.132383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.132318 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:43.134899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.077535 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:43.272045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.255968 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.272045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.256095 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:43.272045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.258682 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.272045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.258797 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:43.425114 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:14:43.326434 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.444s" Oct 30 18:14:43.484997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.472128 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:14:43.512815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.431197 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.513287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.513061 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:43.513287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:43.513182 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:44.058670 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.058588 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.067280 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.067181 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:44.144985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.117269 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.144985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.144913 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.263005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.145770 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:14:44.263005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.147490 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:14:44.284677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.284584 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.286579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.286516 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.294229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.294124 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:44.295043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.294991 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.325262 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.325087 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:44.325740 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.320081 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:44.382527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.378612 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:44.384913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.383427 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.384913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.383526 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:44.536184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.536120 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:14:44.544279 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.544216 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:14:44.783995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.781129 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:14:44.880193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.879516 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:14:44.934236 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:44.934176 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:14:45.086440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.086226 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:14:45.093180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.093121 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.093723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.093669 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 18:14:45.354180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.316851 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.354180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.298509 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.354180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.319409 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:14:45.354180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.316501 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.354180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.342172 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 18:14:45.354180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:45.344796 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:46.047974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:46.047815 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:46.253263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:46.253181 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 18:14:46.254563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:46.254480 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 18:14:47.077976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.077235 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:47.077976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.077375 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.247492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.247416 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:47.277707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.277614 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.342323 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.342108 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:47.367240 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.367118 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:47.383821 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.360024 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:47.409660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.388569 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:14:47.700991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.666819 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:47.945062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.944992 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:47.952567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:47.952409 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:14:48.124332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:48.124223 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:48.124895 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:48.124828 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:48.384790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:48.384664 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:48.386012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:48.385900 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:48.450335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:48.450260 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:48.457277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:48.457206 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:14:49.243973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:49.242206 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:49.340738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:49.340636 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:49.463072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:49.439862 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:49.463072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:49.440074 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:49.681551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:49.681485 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:49.685728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:49.685653 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:14:49.966621 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:14:49.966423 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:14:50.934678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:50.934595 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:52.441060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:52.440954 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:53.847828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.847743 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg"] Oct 30 18:14:53.855483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.855416 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.879151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.879102 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg"] Oct 30 18:14:53.911644 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.911533 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.912046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.911865 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.912215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.912182 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.912640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.912596 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b5fe30-d722-4430-96c1-14044e6450f0-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.912836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.912713 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.912836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.912801 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b5fe30-d722-4430-96c1-14044e6450f0-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.913147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.913057 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.913147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.913105 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tn9x\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-kube-api-access-5tn9x\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.913290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.913177 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:53.978412 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:53.978342 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:54.015565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.015488 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b5fe30-d722-4430-96c1-14044e6450f0-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.015831 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.015922 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tn9x\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-kube-api-access-5tn9x\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.016111 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.016160 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.016217 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016442 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.016410 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.016508 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b5fe30-d722-4430-96c1-14044e6450f0-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.016605 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.016871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.016820 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b5fe30-d722-4430-96c1-14044e6450f0-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.017146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.017110 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.017397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.017351 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.017831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.017795 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.018435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.018388 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.020505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.020399 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.022595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.022526 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b5fe30-d722-4430-96c1-14044e6450f0-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.072096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.071849 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.092106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.092053 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tn9x\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-kube-api-access-5tn9x\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.190272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.190173 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:54.565717 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.565659 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg"] Oct 30 18:14:54.566705 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:14:54.566632 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b5fe30_d722_4430_96c1_14044e6450f0.slice/crio-6a4c8054069bb3b94f5fecaea39659fb9aba9e8504edd15d12dc626633f848c6 WatchSource:0}: Error finding container 6a4c8054069bb3b94f5fecaea39659fb9aba9e8504edd15d12dc626633f848c6: Status 404 returned error can't find the container with id 6a4c8054069bb3b94f5fecaea39659fb9aba9e8504edd15d12dc626633f848c6 Oct 30 18:14:54.572023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.571886 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:54.572338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.572071 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:54.572338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:54.572119 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:14:55.010582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:55.009374 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" event={"ID":"90b5fe30-d722-4430-96c1-14044e6450f0","Type":"ContainerStarted","Data":"845c1500826730553c30064c9c8218d6afcdef7fb149b8112c057b5d5179e596"} Oct 30 18:14:55.010582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:55.009569 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" event={"ID":"90b5fe30-d722-4430-96c1-14044e6450f0","Type":"ContainerStarted","Data":"6a4c8054069bb3b94f5fecaea39659fb9aba9e8504edd15d12dc626633f848c6"} Oct 30 18:14:55.067159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:55.066568 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podStartSLOduration=2.066520368 podStartE2EDuration="2.066520368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:55.052086265 +0000 UTC m=+789.440614383" watchObservedRunningTime="2026-10-30 18:14:55.066520368 +0000 UTC m=+789.455048489" Oct 30 18:14:55.191557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:55.191478 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:14:55.196248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:55.196188 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:55.196639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:55.196328 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:14:55.480755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:55.474502 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:56.203989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:56.199871 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:56.203989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:56.200036 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:14:57.130036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:57.122428 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:57.201205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:57.201127 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:57.201629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:57.201248 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:14:58.201982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:58.200919 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:58.201982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:58.201103 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:14:58.626385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:58.626302 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:14:58.632356 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:58.632263 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:59.193981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:59.192788 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:14:59.193981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:14:59.192925 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:00.260982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.259039 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:00.260982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.259186 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:00.264271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.264165 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f"] Oct 30 18:15:00.324906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.324843 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:00.343751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.343652 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:00.365981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.352086 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc"] Oct 30 18:15:00.381207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.381140 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:15:00.523109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.496451 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e836cff8-aa85-4cb1-ac26-36b0874bba78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f\" (UID: \"e836cff8-aa85-4cb1-ac26-36b0874bba78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:00.659760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.659507 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxl8\" (UniqueName: \"kubernetes.io/projected/e836cff8-aa85-4cb1-ac26-36b0874bba78-kube-api-access-djxl8\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f\" (UID: \"e836cff8-aa85-4cb1-ac26-36b0874bba78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:00.661277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.661181 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:00.781968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.769987 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e836cff8-aa85-4cb1-ac26-36b0874bba78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f\" (UID: \"e836cff8-aa85-4cb1-ac26-36b0874bba78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:00.781968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.770366 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxl8\" (UniqueName: \"kubernetes.io/projected/e836cff8-aa85-4cb1-ac26-36b0874bba78-kube-api-access-djxl8\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f\" (UID: \"e836cff8-aa85-4cb1-ac26-36b0874bba78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:00.833999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:00.822101 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e836cff8-aa85-4cb1-ac26-36b0874bba78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f\" (UID: \"e836cff8-aa85-4cb1-ac26-36b0874bba78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:01.096342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.060074 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwx5\" (UniqueName: \"kubernetes.io/projected/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-kube-api-access-lfwx5\") pod \"maas-api-key-cleanup-29889735-t7xlc\" (UID: \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:01.096342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.060382 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-t7xlc\" (UID: \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:01.276980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.161551 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.276980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.200901 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwx5\" (UniqueName: \"kubernetes.io/projected/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-kube-api-access-lfwx5\") pod \"maas-api-key-cleanup-29889735-t7xlc\" (UID: \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:01.276980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.201246 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-t7xlc\" (UID: \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:01.276980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.209200 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxl8\" (UniqueName: \"kubernetes.io/projected/e836cff8-aa85-4cb1-ac26-36b0874bba78-kube-api-access-djxl8\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f\" (UID: \"e836cff8-aa85-4cb1-ac26-36b0874bba78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:01.276980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.255688 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:01.276980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.255862 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:01.276980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.269139 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889735-t7xlc\" (UID: \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:01.276980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.269327 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.288414 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.304717 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:01.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.316220 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.422983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.373174 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwx5\" (UniqueName: \"kubernetes.io/projected/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-kube-api-access-lfwx5\") pod \"maas-api-key-cleanup-29889735-t7xlc\" (UID: \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:01.599106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.505209 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.604004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.599109 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.650129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.650066 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7"] Oct 30 18:15:01.737995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.690491 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:01.804725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.804625 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.867821 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.867710 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:01.991344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:01.948170 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:02.017607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.017508 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/dddefc02-4b82-4f04-ad2b-a8f42d76724e-kube-api-access-qfv2z\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7\" (UID: \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:02.021812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.021699 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddefc02-4b82-4f04-ad2b-a8f42d76724e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7\" (UID: \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:02.300667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.291673 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:02.300667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.291822 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:02.302469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.302355 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:02.382621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.382512 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:15:02.506159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.506077 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:15:02.558789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.558586 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddefc02-4b82-4f04-ad2b-a8f42d76724e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7\" (UID: \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:02.629244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.629150 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/dddefc02-4b82-4f04-ad2b-a8f42d76724e-kube-api-access-qfv2z\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7\" (UID: \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:02.725987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.585601 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddefc02-4b82-4f04-ad2b-a8f42d76724e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7\" (UID: \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:02.725987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.404537 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:02.899029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:02.893174 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:15:03.336115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.325362 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:15:03.382614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.382527 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:03.503480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.503390 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:15:03.583458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.583347 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:03.643135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.618692 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:03.770002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.751602 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:15:03.818018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.802069 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:03.818847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.814760 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:03.936255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:03.918114 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:04.023008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:04.012566 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/dddefc02-4b82-4f04-ad2b-a8f42d76724e-kube-api-access-qfv2z\") pod \"maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7\" (UID: \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:04.175045 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:04.174877 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.242s" Oct 30 18:15:04.223858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:04.223677 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:04.622413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:04.622310 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:04.822706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:04.822603 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:06.551666 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.551476 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" podUID="4326dbaf-621c-4d40-9f86-e29dbdd1b649" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.55:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:06.552717 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:04.996708 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:04.996799 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:04.898607 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.649907 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:05.249783 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:05.524639 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.031631 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.043167 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.296332 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:05.032125 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:05.249648 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.756382 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.649358 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.791141 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.839172 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.839281 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.947315 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.947480 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.010501 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.096885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:06.406894 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 18:15:07.164350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.154165 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.421009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.410269 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.514365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.487401 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:07.599258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.574650 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.61:50052\" within 1s: context deadline exceeded" Oct 30 18:15:07.599258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.579196 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.728980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.679335 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.785980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.682886 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:07.935819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.755413 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.935819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.775642 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:15:07.935819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.778077 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.951657 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:15:07.782613 5624 logging.go:55] [core] [Channel #427 SubChannel #428]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.61:50052", ServerName: "10.134.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.61:50052: operation was canceled" Oct 30 18:15:07.951657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.784302 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:07.955335 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:07.908974 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.725s" Oct 30 18:15:07.955335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.955022 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f"] Oct 30 18:15:07.955335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.955212 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:15:07.963480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.963408 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:07.964011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:07.963959 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:08.058070 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.057552 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:15:08.136345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.135392 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc"] Oct 30 18:15:08.151563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151324 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7"] Oct 30 18:15:08.151563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151526 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151588 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151630 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151688 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151720 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f"] Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151761 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc"] Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151824 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151888 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.151967 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.152026 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.152080 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:15:08.152108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.152122 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:15:08.152606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.152166 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:15:08.152606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.152242 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:15:08.152606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.152284 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:15:08.152606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.152322 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:15:08.152606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.152364 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:15:08.177232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.165907 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:08.351448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.351246 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 18:15:08.352011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.351913 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 18:15:08.673453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.673374 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7"] Oct 30 18:15:08.820834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.820069 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z"] Oct 30 18:15:08.840234 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.840160 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z"] Oct 30 18:15:08.862414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:08.841299 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.052598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.052510 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.052598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.052612 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.053136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.052720 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d96999-4d59-46f9-b7de-f4349a071b6c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.053136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.052975 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.053272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.053182 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.053339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.053299 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.053523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.053454 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.053653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.053536 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7w5x\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-kube-api-access-w7w5x\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.053653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.053581 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.058948 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.058861 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" event={"ID":"e836cff8-aa85-4cb1-ac26-36b0874bba78","Type":"ContainerStarted","Data":"ae94f876dc7af6b11e4279970973f1ef9325aaf6f4b5ac961fe60b0ad35298fa"} Oct 30 18:15:09.064921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.064840 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" event={"ID":"dddefc02-4b82-4f04-ad2b-a8f42d76724e","Type":"ContainerStarted","Data":"5b7723deac3a505d235381745df048c8e6a339fb0d5b47ba7fe75c969fc58bc8"} Oct 30 18:15:09.071060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.070977 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" event={"ID":"a3abc77b-667b-4af2-bc8a-9f0e4b00889d","Type":"ContainerStarted","Data":"502056058d5ba16ac954919d258b41e100b618de1202032063e4449ef43f5c72"} Oct 30 18:15:09.071437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.071102 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" event={"ID":"a3abc77b-667b-4af2-bc8a-9f0e4b00889d","Type":"ContainerStarted","Data":"e26a346d0be539f75e47bbae469de7c13162bfc6324fec035ccdb64281d7a886"} Oct 30 18:15:09.126496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.126325 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" podStartSLOduration=9.126277417 podStartE2EDuration="9.126277417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:09.113105932 +0000 UTC m=+803.501634050" watchObservedRunningTime="2026-10-30 18:15:09.126277417 +0000 UTC m=+803.514805535" Oct 30 18:15:09.158700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.158588 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d96999-4d59-46f9-b7de-f4349a071b6c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.159264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.158922 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.159264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.159203 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.159444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.159323 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.159498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.159456 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.159563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.159538 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7w5x\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-kube-api-access-w7w5x\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.159624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.159584 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.160724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.159784 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.160724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.159854 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.160724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.160190 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.160724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.160220 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.161209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.160753 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.161209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.160912 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.161436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.161350 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d96999-4d59-46f9-b7de-f4349a071b6c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.165019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.164769 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.165019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.164995 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.185047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.183977 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.185047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.184824 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7w5x\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-kube-api-access-w7w5x\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.191061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.190506 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:15:09.263862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.263544 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:09.722427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.718981 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z"] Oct 30 18:15:09.743080 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:15:09.743011 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d96999_4d59_46f9_b7de_f4349a071b6c.slice/crio-9889e2490ea171bf9555ddb211066efced14ed8443e6d7c8a35b1dbed8f60dc6 WatchSource:0}: Error finding container 9889e2490ea171bf9555ddb211066efced14ed8443e6d7c8a35b1dbed8f60dc6: Status 404 returned error can't find the container with id 9889e2490ea171bf9555ddb211066efced14ed8443e6d7c8a35b1dbed8f60dc6 Oct 30 18:15:09.750080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.750002 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:09.750432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.750154 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:09.750432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.750205 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:09.822502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.822291 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:15:09.840721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.840626 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:09.949136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:09.948338 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:10.271980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.262056 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" event={"ID":"b5d96999-4d59-46f9-b7de-f4349a071b6c","Type":"ContainerStarted","Data":"578e4a7f2a7536419ccbdf6bc3ba621df02e50814c12580e545de90929dbd634"} Oct 30 18:15:10.271980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.262191 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" event={"ID":"b5d96999-4d59-46f9-b7de-f4349a071b6c","Type":"ContainerStarted","Data":"9889e2490ea171bf9555ddb211066efced14ed8443e6d7c8a35b1dbed8f60dc6"} Oct 30 18:15:10.290096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.278676 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:10.393041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.350849 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:10.393041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.350983 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:10.393041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.356477 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podStartSLOduration=2.356435773 podStartE2EDuration="2.356435773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:10.352022254 +0000 UTC m=+804.740550373" watchObservedRunningTime="2026-10-30 18:15:10.356435773 +0000 UTC m=+804.744963893" Oct 30 18:15:10.425076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.406125 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" event={"ID":"dddefc02-4b82-4f04-ad2b-a8f42d76724e","Type":"ContainerStarted","Data":"173d0700bf7972c4bf3478e48b256cbd74c7e533c7c6fe5c518ec3c9b083e920"} Oct 30 18:15:10.431589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.430498 5624 generic.go:356] "Generic (PLEG): container finished" podID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerID="502056058d5ba16ac954919d258b41e100b618de1202032063e4449ef43f5c72" exitCode=60 Oct 30 18:15:10.431589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.430928 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" event={"ID":"a3abc77b-667b-4af2-bc8a-9f0e4b00889d","Type":"ContainerDied","Data":"502056058d5ba16ac954919d258b41e100b618de1202032063e4449ef43f5c72"} Oct 30 18:15:10.454528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.454182 5624 scope.go:122] "RemoveContainer" containerID="502056058d5ba16ac954919d258b41e100b618de1202032063e4449ef43f5c72" Oct 30 18:15:10.457221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.457131 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" event={"ID":"e836cff8-aa85-4cb1-ac26-36b0874bba78","Type":"ContainerStarted","Data":"8958278c271a28eaab55e8c04cf5275dd1fc19e78061b3c3d23063569dee096e"} Oct 30 18:15:10.478570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.478092 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:15:10.492501 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.492355 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" podStartSLOduration=10.492313097 podStartE2EDuration="10.492313097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:10.460429596 +0000 UTC m=+804.848957714" watchObservedRunningTime="2026-10-30 18:15:10.492313097 +0000 UTC m=+804.880841217" Oct 30 18:15:10.580312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:10.580169 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" podStartSLOduration=10.580122755 podStartE2EDuration="10.580122755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:10.55481783 +0000 UTC m=+804.943346034" watchObservedRunningTime="2026-10-30 18:15:10.580122755 +0000 UTC m=+804.968650873" Oct 30 18:15:11.264382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.264317 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:11.265074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.264420 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:11.462900 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.462710 5624 generic.go:356] "Generic (PLEG): container finished" podID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerID="173d0700bf7972c4bf3478e48b256cbd74c7e533c7c6fe5c518ec3c9b083e920" exitCode=60 Oct 30 18:15:11.462900 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.462805 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" event={"ID":"dddefc02-4b82-4f04-ad2b-a8f42d76724e","Type":"ContainerDied","Data":"173d0700bf7972c4bf3478e48b256cbd74c7e533c7c6fe5c518ec3c9b083e920"} Oct 30 18:15:11.463579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.463541 5624 scope.go:122] "RemoveContainer" containerID="173d0700bf7972c4bf3478e48b256cbd74c7e533c7c6fe5c518ec3c9b083e920" Oct 30 18:15:11.467888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.467813 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" event={"ID":"a3abc77b-667b-4af2-bc8a-9f0e4b00889d","Type":"ContainerStarted","Data":"3eb491d78fc866a806a171cb98ee3757b720212f68d613cec69c5cf5c8f674db"} Oct 30 18:15:11.471708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.471656 5624 generic.go:356] "Generic (PLEG): container finished" podID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerID="8958278c271a28eaab55e8c04cf5275dd1fc19e78061b3c3d23063569dee096e" exitCode=60 Oct 30 18:15:11.471851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.471756 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" event={"ID":"e836cff8-aa85-4cb1-ac26-36b0874bba78","Type":"ContainerDied","Data":"8958278c271a28eaab55e8c04cf5275dd1fc19e78061b3c3d23063569dee096e"} Oct 30 18:15:11.472615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.472571 5624 scope.go:122] "RemoveContainer" containerID="8958278c271a28eaab55e8c04cf5275dd1fc19e78061b3c3d23063569dee096e" Oct 30 18:15:11.821100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:11.820081 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:12.264721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:12.264650 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:12.265457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:12.264752 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:12.526297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:12.526089 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" event={"ID":"dddefc02-4b82-4f04-ad2b-a8f42d76724e","Type":"ContainerStarted","Data":"3e50a54cbdf459d87dbc2f0c1eb29e693971944754c0d66b37a40b5423122604"} Oct 30 18:15:12.533389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:12.533308 5624 generic.go:356] "Generic (PLEG): container finished" podID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerID="3eb491d78fc866a806a171cb98ee3757b720212f68d613cec69c5cf5c8f674db" exitCode=60 Oct 30 18:15:12.534705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:12.533955 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" event={"ID":"a3abc77b-667b-4af2-bc8a-9f0e4b00889d","Type":"ContainerDied","Data":"3eb491d78fc866a806a171cb98ee3757b720212f68d613cec69c5cf5c8f674db"} Oct 30 18:15:12.535081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:12.535024 5624 scope.go:122] "RemoveContainer" containerID="502056058d5ba16ac954919d258b41e100b618de1202032063e4449ef43f5c72" Oct 30 18:15:12.536079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:12.536037 5624 scope.go:122] "RemoveContainer" containerID="3eb491d78fc866a806a171cb98ee3757b720212f68d613cec69c5cf5c8f674db" Oct 30 18:15:12.536837 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:12.536792 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-t7xlc_odh-ai-gateway-infra(a3abc77b-667b-4af2-bc8a-9f0e4b00889d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" Oct 30 18:15:12.543530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:12.543442 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" event={"ID":"e836cff8-aa85-4cb1-ac26-36b0874bba78","Type":"ContainerStarted","Data":"17781fab554fbeced734b40686dbc72980d2da9947b3160fd3a493133ddb238a"} Oct 30 18:15:13.265347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.265280 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:13.266193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.265389 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:13.577995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.577870 5624 generic.go:356] "Generic (PLEG): container finished" podID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerID="3e50a54cbdf459d87dbc2f0c1eb29e693971944754c0d66b37a40b5423122604" exitCode=60 Oct 30 18:15:13.578340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.577983 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" event={"ID":"dddefc02-4b82-4f04-ad2b-a8f42d76724e","Type":"ContainerDied","Data":"3e50a54cbdf459d87dbc2f0c1eb29e693971944754c0d66b37a40b5423122604"} Oct 30 18:15:13.578340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.578138 5624 scope.go:122] "RemoveContainer" containerID="173d0700bf7972c4bf3478e48b256cbd74c7e533c7c6fe5c518ec3c9b083e920" Oct 30 18:15:13.578783 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.578745 5624 scope.go:122] "RemoveContainer" containerID="3e50a54cbdf459d87dbc2f0c1eb29e693971944754c0d66b37a40b5423122604" Oct 30 18:15:13.579620 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:13.579567 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7_odh-ai-gateway-infra(dddefc02-4b82-4f04-ad2b-a8f42d76724e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" Oct 30 18:15:13.604981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.585391 5624 scope.go:122] "RemoveContainer" containerID="3eb491d78fc866a806a171cb98ee3757b720212f68d613cec69c5cf5c8f674db" Oct 30 18:15:13.604981 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:13.586361 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-t7xlc_odh-ai-gateway-infra(a3abc77b-667b-4af2-bc8a-9f0e4b00889d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" Oct 30 18:15:13.649978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.636179 5624 generic.go:356] "Generic (PLEG): container finished" podID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerID="17781fab554fbeced734b40686dbc72980d2da9947b3160fd3a493133ddb238a" exitCode=60 Oct 30 18:15:13.649978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.636516 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" event={"ID":"e836cff8-aa85-4cb1-ac26-36b0874bba78","Type":"ContainerDied","Data":"17781fab554fbeced734b40686dbc72980d2da9947b3160fd3a493133ddb238a"} Oct 30 18:15:13.649978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.646894 5624 scope.go:122] "RemoveContainer" containerID="17781fab554fbeced734b40686dbc72980d2da9947b3160fd3a493133ddb238a" Oct 30 18:15:13.649978 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:13.647989 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f_odh-ai-gateway-infra(e836cff8-aa85-4cb1-ac26-36b0874bba78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" Oct 30 18:15:13.683511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.683474 5624 scope.go:122] "RemoveContainer" containerID="8958278c271a28eaab55e8c04cf5275dd1fc19e78061b3c3d23063569dee096e" Oct 30 18:15:13.735770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:13.735706 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:14.264724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.264647 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:14.265085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.264766 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:14.357300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.357208 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt"] Oct 30 18:15:14.367396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.367336 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.417042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.416169 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.418343 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7c7d\" (UniqueName: \"kubernetes.io/projected/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-kube-api-access-l7c7d\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.418602 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tmp-dir\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.418699 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-home\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.418973 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-dshm\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.419059 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-model-cache\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.419435 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tls-certs\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.420128 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"kube-root-ca.crt\"" Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.420376 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"distinct-w1-fb2fa0-kserve-self-signed-certs\"" Oct 30 18:15:14.441994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.420547 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"default-dockercfg-zff9h\"" Oct 30 18:15:14.447203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.447129 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"openshift-service-ca.crt\"" Oct 30 18:15:14.449172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.449123 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt"] Oct 30 18:15:14.522282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.522091 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tls-certs\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.522282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.522238 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7c7d\" (UniqueName: \"kubernetes.io/projected/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-kube-api-access-l7c7d\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.522673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.522420 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tmp-dir\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.522673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.522501 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-home\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.524649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.524582 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-dshm\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.525077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.524718 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-home\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.526737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.526664 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-model-cache\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.528143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.528080 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tmp-dir\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.528715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.528675 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-model-cache\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.530921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.530547 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tls-certs\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.532470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.532402 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-dshm\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.555434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.555347 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7c7d\" (UniqueName: \"kubernetes.io/projected/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-kube-api-access-l7c7d\") pod \"distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:14.799593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:14.790590 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:15.157971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:15.157865 5624 scope.go:122] "RemoveContainer" containerID="3e50a54cbdf459d87dbc2f0c1eb29e693971944754c0d66b37a40b5423122604" Oct 30 18:15:15.168998 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:15.160544 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7_odh-ai-gateway-infra(dddefc02-4b82-4f04-ad2b-a8f42d76724e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" Oct 30 18:15:15.210797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:15.210696 5624 scope.go:122] "RemoveContainer" containerID="17781fab554fbeced734b40686dbc72980d2da9947b3160fd3a493133ddb238a" Oct 30 18:15:15.215012 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:15.211611 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f_odh-ai-gateway-infra(e836cff8-aa85-4cb1-ac26-36b0874bba78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" Oct 30 18:15:15.273981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:15.265068 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:15.273981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:15.265169 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:15.745952 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:15.745844 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt"] Oct 30 18:15:15.747867 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:15:15.747790 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a99c9dc_0e26_42a4_8d0f_105b71066bc2.slice/crio-c88f1e0e8db5f0558ab173b338c7bf2dce87d8766b74f65513a46748a5e25a51 WatchSource:0}: Error finding container c88f1e0e8db5f0558ab173b338c7bf2dce87d8766b74f65513a46748a5e25a51: Status 404 returned error can't find the container with id c88f1e0e8db5f0558ab173b338c7bf2dce87d8766b74f65513a46748a5e25a51 Oct 30 18:15:15.822005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:15.821127 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:16.241664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:16.241571 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" event={"ID":"6a99c9dc-0e26-42a4-8d0f-105b71066bc2","Type":"ContainerStarted","Data":"71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f"} Oct 30 18:15:16.241664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:16.241665 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" event={"ID":"6a99c9dc-0e26-42a4-8d0f-105b71066bc2","Type":"ContainerStarted","Data":"c88f1e0e8db5f0558ab173b338c7bf2dce87d8766b74f65513a46748a5e25a51"} Oct 30 18:15:16.265046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:16.264909 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:16.265486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:16.265082 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:16.283854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:16.283621 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podStartSLOduration=2.283517817 podStartE2EDuration="2.283517817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:16.282291606 +0000 UTC m=+810.670819722" watchObservedRunningTime="2026-10-30 18:15:16.283517817 +0000 UTC m=+810.672045936" Oct 30 18:15:16.813992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:16.806403 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:17.266867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:17.266724 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:17.267284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:17.266897 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:17.546757 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:17.546580 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:18.034532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:18.032437 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:18.288097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:18.286568 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:18.288097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:18.286891 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:19.265371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:19.265284 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:19.266327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:19.265415 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 18:15:19.336584 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:19.336506 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:19.993659 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:19.993490 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:15:20.412977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:20.412884 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:20.414080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:20.414024 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:20.416816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:20.416733 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:15:21.053193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:21.053115 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:22.741912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:22.741851 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:23.802043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.801875 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4"] Oct 30 18:15:23.820162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.819984 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.836083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.836011 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4"] Oct 30 18:15:23.871099 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.871013 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.871653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.871325 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.871653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.871477 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.871653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.871523 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.871653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.871620 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.872768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.872048 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcb9\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-kube-api-access-hkcb9\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.872768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.872156 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.872768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.872285 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.872768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.872606 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.975345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.975260 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.975345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.975343 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.975810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.975408 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.975810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.975624 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcb9\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-kube-api-access-hkcb9\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.975810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.975683 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.975810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.975783 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.976085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.975922 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.976191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.976138 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.976380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.976323 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.976623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.976394 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.976623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.976427 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.976623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.976581 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.976790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.976690 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.977725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.977669 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.979394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.979334 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.979711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.979677 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.998772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.998697 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:23.999174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:23.999079 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcb9\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-kube-api-access-hkcb9\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:24.172390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.172298 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:24.350394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.350317 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:24.558075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.558010 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4"] Oct 30 18:15:24.560284 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:15:24.560214 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ebbf0d_9cfe_452d_b066_5162a4ebfff6.slice/crio-a553f67c4f5fb97b86c0154036831bdedcd4ecad714fa1a51aa475436b09ab94 WatchSource:0}: Error finding container a553f67c4f5fb97b86c0154036831bdedcd4ecad714fa1a51aa475436b09ab94: Status 404 returned error can't find the container with id a553f67c4f5fb97b86c0154036831bdedcd4ecad714fa1a51aa475436b09ab94 Oct 30 18:15:24.567052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.566856 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:24.567052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.567063 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:24.567477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.567149 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:24.644575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.644480 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" event={"ID":"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6","Type":"ContainerStarted","Data":"a553f67c4f5fb97b86c0154036831bdedcd4ecad714fa1a51aa475436b09ab94"} Oct 30 18:15:24.784833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.784754 5624 scope.go:122] "RemoveContainer" containerID="3e50a54cbdf459d87dbc2f0c1eb29e693971944754c0d66b37a40b5423122604" Oct 30 18:15:24.802094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.801763 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:24.824612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:24.824467 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:25.445355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.445283 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2"] Oct 30 18:15:25.455329 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.454899 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.471455 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.471375 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2"] Oct 30 18:15:25.477726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.477653 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"distinct-2-w0-6ce5eb-kserve-self-signed-certs\"" Oct 30 18:15:25.525307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.525219 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:25.535695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.533680 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2svk\" (UniqueName: \"kubernetes.io/projected/f0d814b1-9303-4d63-804f-0487021f295b-kube-api-access-j2svk\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.535695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.533837 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-model-cache\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.535695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.534206 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-home\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.535695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.534464 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-dshm\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.535695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.534565 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d814b1-9303-4d63-804f-0487021f295b-tls-certs\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.535695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.534636 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-tmp-dir\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.637055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.636954 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-dshm\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.637667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.637630 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d814b1-9303-4d63-804f-0487021f295b-tls-certs\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.637908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.637890 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-tmp-dir\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.638248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.638222 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2svk\" (UniqueName: \"kubernetes.io/projected/f0d814b1-9303-4d63-804f-0487021f295b-kube-api-access-j2svk\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.638413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.638394 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-model-cache\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.638894 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.638870 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-home\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.639240 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.638456 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-tmp-dir\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.639625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.639578 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-model-cache\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.640120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.640088 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-home\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.684152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.679104 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d814b1-9303-4d63-804f-0487021f295b-tls-certs\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.907972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.903206 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2svk\" (UniqueName: \"kubernetes.io/projected/f0d814b1-9303-4d63-804f-0487021f295b-kube-api-access-j2svk\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:25.907972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:25.906719 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-dshm\") pod \"distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:26.031594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.031365 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podStartSLOduration=3.031329096 podStartE2EDuration="3.031329096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:26.026567581 +0000 UTC m=+820.415095700" watchObservedRunningTime="2026-10-30 18:15:26.031329096 +0000 UTC m=+820.419857214" Oct 30 18:15:26.048374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.048284 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" event={"ID":"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6","Type":"ContainerStarted","Data":"5b7328890d765074b740d3cd0beb7616f1b3c2afa1f2c766f5fb6d74c256dbf7"} Oct 30 18:15:26.048374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.048380 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" event={"ID":"dddefc02-4b82-4f04-ad2b-a8f42d76724e","Type":"ContainerStarted","Data":"ad27159433f1c099bfeed13ea754ed5619676736010ecc52bbb04ac4ecae87c5"} Oct 30 18:15:26.048836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.048427 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:26.049636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.049443 5624 scope.go:122] "RemoveContainer" containerID="3eb491d78fc866a806a171cb98ee3757b720212f68d613cec69c5cf5c8f674db" Oct 30 18:15:26.154869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.154813 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:26.173297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.173134 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:26.236054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.185224 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.236054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.185369 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:15:26.239657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.239417 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:26.836555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:26.836029 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2"] Oct 30 18:15:27.065069 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.064993 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" event={"ID":"f0d814b1-9303-4d63-804f-0487021f295b","Type":"ContainerStarted","Data":"5c8055f2da9aa3dd7dc4dc9fd2ed29a793b6135a0968f0902914874824b00cf7"} Oct 30 18:15:27.074752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.072266 5624 generic.go:356] "Generic (PLEG): container finished" podID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerID="ad27159433f1c099bfeed13ea754ed5619676736010ecc52bbb04ac4ecae87c5" exitCode=60 Oct 30 18:15:27.074752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.072346 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" event={"ID":"dddefc02-4b82-4f04-ad2b-a8f42d76724e","Type":"ContainerDied","Data":"ad27159433f1c099bfeed13ea754ed5619676736010ecc52bbb04ac4ecae87c5"} Oct 30 18:15:27.074752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.072469 5624 scope.go:122] "RemoveContainer" containerID="3e50a54cbdf459d87dbc2f0c1eb29e693971944754c0d66b37a40b5423122604" Oct 30 18:15:27.074752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.073549 5624 scope.go:122] "RemoveContainer" containerID="ad27159433f1c099bfeed13ea754ed5619676736010ecc52bbb04ac4ecae87c5" Oct 30 18:15:27.080140 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:27.080073 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7_odh-ai-gateway-infra(dddefc02-4b82-4f04-ad2b-a8f42d76724e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" Oct 30 18:15:27.141685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.141560 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7"] Oct 30 18:15:27.173493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.173430 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.173815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.173532 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:15:27.844364 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.844286 5624 scope.go:122] "RemoveContainer" containerID="17781fab554fbeced734b40686dbc72980d2da9947b3160fd3a493133ddb238a" Oct 30 18:15:27.964229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:27.964152 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:28.177731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.174438 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.177731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.174553 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:15:28.230896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.230808 5624 generic.go:356] "Generic (PLEG): container finished" podID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerID="884d66d73384de9611fe8f8c0bf646587ce9196d3860e5f10c39db7062093ff8" exitCode=60 Oct 30 18:15:28.232217 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.232147 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" event={"ID":"a3abc77b-667b-4af2-bc8a-9f0e4b00889d","Type":"ContainerDied","Data":"884d66d73384de9611fe8f8c0bf646587ce9196d3860e5f10c39db7062093ff8"} Oct 30 18:15:28.232719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.232681 5624 scope.go:122] "RemoveContainer" containerID="3eb491d78fc866a806a171cb98ee3757b720212f68d613cec69c5cf5c8f674db" Oct 30 18:15:28.234556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.234496 5624 scope.go:122] "RemoveContainer" containerID="884d66d73384de9611fe8f8c0bf646587ce9196d3860e5f10c39db7062093ff8" Oct 30 18:15:28.235446 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:28.235385 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889735-t7xlc_odh-ai-gateway-infra(a3abc77b-667b-4af2-bc8a-9f0e4b00889d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" Oct 30 18:15:28.238730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.238645 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" event={"ID":"f0d814b1-9303-4d63-804f-0487021f295b","Type":"ContainerStarted","Data":"215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f"} Oct 30 18:15:28.329227 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.329011 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podStartSLOduration=3.328959958 podStartE2EDuration="3.328959958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:28.31697709 +0000 UTC m=+822.705505209" watchObservedRunningTime="2026-10-30 18:15:28.328959958 +0000 UTC m=+822.717488072" Oct 30 18:15:28.843387 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.843307 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:28.962874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.962803 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddefc02-4b82-4f04-ad2b-a8f42d76724e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddefc02-4b82-4f04-ad2b-a8f42d76724e-maas-api-service-ca\") pod \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\" (UID: \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\") " Oct 30 18:15:28.963252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.963148 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddefc02-4b82-4f04-ad2b-a8f42d76724e-kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/dddefc02-4b82-4f04-ad2b-a8f42d76724e-kube-api-access-qfv2z\") pod \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\" (UID: \"dddefc02-4b82-4f04-ad2b-a8f42d76724e\") " Oct 30 18:15:28.964597 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.964524 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddefc02-4b82-4f04-ad2b-a8f42d76724e-maas-api-service-ca" pod "dddefc02-4b82-4f04-ad2b-a8f42d76724e" (UID: "dddefc02-4b82-4f04-ad2b-a8f42d76724e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:28.965762 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.965712 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddefc02-4b82-4f04-ad2b-a8f42d76724e-maas-api-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:15:28.968842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:28.968769 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddefc02-4b82-4f04-ad2b-a8f42d76724e-kube-api-access-qfv2z" pod "dddefc02-4b82-4f04-ad2b-a8f42d76724e" (UID: "dddefc02-4b82-4f04-ad2b-a8f42d76724e"). InnerVolumeSpecName "kube-api-access-qfv2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:29.068543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.068352 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfv2z\" (UniqueName: \"kubernetes.io/projected/dddefc02-4b82-4f04-ad2b-a8f42d76724e-kube-api-access-qfv2z\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:15:29.156483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.156396 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:29.173822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.173759 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.174225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.174134 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:15:29.253047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.252992 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" Oct 30 18:15:29.253741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.253045 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7" event={"ID":"dddefc02-4b82-4f04-ad2b-a8f42d76724e","Type":"ContainerDied","Data":"5b7723deac3a505d235381745df048c8e6a339fb0d5b47ba7fe75c969fc58bc8"} Oct 30 18:15:29.253741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.253147 5624 scope.go:122] "RemoveContainer" containerID="ad27159433f1c099bfeed13ea754ed5619676736010ecc52bbb04ac4ecae87c5" Oct 30 18:15:29.263600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.261765 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" event={"ID":"e836cff8-aa85-4cb1-ac26-36b0874bba78","Type":"ContainerStarted","Data":"5383c77130962ba464d96f4cc891dc9c58e5b1ef7e23223eb7bc784c952d2e3f"} Oct 30 18:15:29.371221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.370990 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7"] Oct 30 18:15:29.379924 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.379841 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc"] Oct 30 18:15:29.434960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.431063 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8b6cf1-29889735-957g7"] Oct 30 18:15:29.639083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.638844 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:29.761092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:29.761046 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:30.052425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.052356 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" path="/var/lib/kubelet/pods/dddefc02-4b82-4f04-ad2b-a8f42d76724e/volumes" Oct 30 18:15:30.058634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.058058 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:30.173365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.173194 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.173365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.173335 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:15:30.227282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.227198 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-maas-api-service-ca\") pod \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\" (UID: \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\") " Oct 30 18:15:30.227625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.227562 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-kube-api-access-lfwx5\" (UniqueName: \"kubernetes.io/projected/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-kube-api-access-lfwx5\") pod \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\" (UID: \"a3abc77b-667b-4af2-bc8a-9f0e4b00889d\") " Oct 30 18:15:30.231197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.231111 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-maas-api-service-ca" pod "a3abc77b-667b-4af2-bc8a-9f0e4b00889d" (UID: "a3abc77b-667b-4af2-bc8a-9f0e4b00889d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:30.236352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.236263 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-kube-api-access-lfwx5" pod "a3abc77b-667b-4af2-bc8a-9f0e4b00889d" (UID: "a3abc77b-667b-4af2-bc8a-9f0e4b00889d"). InnerVolumeSpecName "kube-api-access-lfwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:30.283402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.283347 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" Oct 30 18:15:30.284137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.283346 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc" event={"ID":"a3abc77b-667b-4af2-bc8a-9f0e4b00889d","Type":"ContainerDied","Data":"e26a346d0be539f75e47bbae469de7c13162bfc6324fec035ccdb64281d7a886"} Oct 30 18:15:30.284137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.283554 5624 scope.go:122] "RemoveContainer" containerID="884d66d73384de9611fe8f8c0bf646587ce9196d3860e5f10c39db7062093ff8" Oct 30 18:15:30.332596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.332508 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfwx5\" (UniqueName: \"kubernetes.io/projected/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-kube-api-access-lfwx5\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:15:30.332596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.332574 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3abc77b-667b-4af2-bc8a-9f0e4b00889d-maas-api-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:15:30.336107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.335444 5624 generic.go:356] "Generic (PLEG): container finished" podID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerID="5383c77130962ba464d96f4cc891dc9c58e5b1ef7e23223eb7bc784c952d2e3f" exitCode=60 Oct 30 18:15:30.336107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.336002 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" event={"ID":"e836cff8-aa85-4cb1-ac26-36b0874bba78","Type":"ContainerDied","Data":"5383c77130962ba464d96f4cc891dc9c58e5b1ef7e23223eb7bc784c952d2e3f"} Oct 30 18:15:30.338599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.338529 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f"] Oct 30 18:15:30.340145 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.340078 5624 scope.go:122] "RemoveContainer" containerID="5383c77130962ba464d96f4cc891dc9c58e5b1ef7e23223eb7bc784c952d2e3f" Oct 30 18:15:30.341977 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:30.341148 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f_odh-ai-gateway-infra(e836cff8-aa85-4cb1-ac26-36b0874bba78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" Oct 30 18:15:30.355834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.355779 5624 scope.go:122] "RemoveContainer" containerID="17781fab554fbeced734b40686dbc72980d2da9947b3160fd3a493133ddb238a" Oct 30 18:15:30.377462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.377387 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc"] Oct 30 18:15:30.383442 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:30.383385 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889735-t7xlc"] Oct 30 18:15:31.174067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:31.173917 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:15:31.174536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:31.174123 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:15:31.433144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:31.432841 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:31.678969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:31.674601 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:31.855373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:31.855308 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" path="/var/lib/kubelet/pods/a3abc77b-667b-4af2-bc8a-9f0e4b00889d/volumes" Oct 30 18:15:31.938600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:31.938497 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:32.052256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.051542 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e836cff8-aa85-4cb1-ac26-36b0874bba78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e836cff8-aa85-4cb1-ac26-36b0874bba78-maas-api-service-ca\") pod \"e836cff8-aa85-4cb1-ac26-36b0874bba78\" (UID: \"e836cff8-aa85-4cb1-ac26-36b0874bba78\") " Oct 30 18:15:32.052256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.051687 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e836cff8-aa85-4cb1-ac26-36b0874bba78-kube-api-access-djxl8\" (UniqueName: \"kubernetes.io/projected/e836cff8-aa85-4cb1-ac26-36b0874bba78-kube-api-access-djxl8\") pod \"e836cff8-aa85-4cb1-ac26-36b0874bba78\" (UID: \"e836cff8-aa85-4cb1-ac26-36b0874bba78\") " Oct 30 18:15:32.055166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.055093 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e836cff8-aa85-4cb1-ac26-36b0874bba78-maas-api-service-ca" pod "e836cff8-aa85-4cb1-ac26-36b0874bba78" (UID: "e836cff8-aa85-4cb1-ac26-36b0874bba78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:32.061469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.061356 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e836cff8-aa85-4cb1-ac26-36b0874bba78-kube-api-access-djxl8" pod "e836cff8-aa85-4cb1-ac26-36b0874bba78" (UID: "e836cff8-aa85-4cb1-ac26-36b0874bba78"). InnerVolumeSpecName "kube-api-access-djxl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:32.158031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.157524 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e836cff8-aa85-4cb1-ac26-36b0874bba78-maas-api-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.158031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.157592 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djxl8\" (UniqueName: \"kubernetes.io/projected/e836cff8-aa85-4cb1-ac26-36b0874bba78-kube-api-access-djxl8\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.174391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.174306 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:15:32.174761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.174443 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:15:32.388988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.388499 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" event={"ID":"e836cff8-aa85-4cb1-ac26-36b0874bba78","Type":"ContainerDied","Data":"ae94f876dc7af6b11e4279970973f1ef9325aaf6f4b5ac961fe60b0ad35298fa"} Oct 30 18:15:32.388988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.388667 5624 scope.go:122] "RemoveContainer" containerID="5383c77130962ba464d96f4cc891dc9c58e5b1ef7e23223eb7bc784c952d2e3f" Oct 30 18:15:32.389460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.389060 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f" Oct 30 18:15:32.460802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.460717 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f"] Oct 30 18:15:32.475177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:32.475102 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889735-w426f"] Oct 30 18:15:33.030655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:33.030578 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:33.175994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:33.175604 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 18:15:33.175994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:33.175957 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 18:15:33.805057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:33.804959 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" path="/var/lib/kubelet/pods/e836cff8-aa85-4cb1-ac26-36b0874bba78/volumes" Oct 30 18:15:34.181889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:34.181820 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:34.182417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:34.182374 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:34.184920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:34.184852 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:15:34.627572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:34.627503 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:36.155976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:36.155890 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:36.173756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:36.173683 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:36.240949 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:36.240874 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:36.469571 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:36.469413 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:37.795887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:37.795806 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:38.080889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.080655 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:38.159919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.159848 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:15:38.798304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.798225 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9"] Oct 30 18:15:38.803638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.803551 5624 cpu_manager.go:414] "containerMap: removing container" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:38.808990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.804086 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:38.809481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809436 5624 cpu_manager.go:414] "containerMap: removing container" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:38.809653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809615 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:38.809856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809675 5624 cpu_manager.go:414] "containerMap: removing container" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:38.809856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809686 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:38.809856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809769 5624 cpu_manager.go:414] "containerMap: removing container" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:38.809856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809779 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:38.809856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809808 5624 cpu_manager.go:414] "containerMap: removing container" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:38.809856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809819 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:38.810416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809874 5624 cpu_manager.go:414] "containerMap: removing container" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:38.810416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809883 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:38.810416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809970 5624 cpu_manager.go:414] "containerMap: removing container" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:38.810416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.809981 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:38.810416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.810367 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:38.810416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.810395 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:38.810416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.810419 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:38.810883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.810431 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:38.810883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.810455 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:38.810883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.810469 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:38.837169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.836114 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.840381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.840284 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9"] Oct 30 18:15:38.964281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.964202 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.964591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.964452 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.964797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.964764 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.964974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.964891 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.965062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.964976 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.965525 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.965492 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.965603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.965554 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjxd\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-kube-api-access-cpjxd\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.965826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.965786 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:38.966023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:38.965906 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.069562 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.069372 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.069562 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.069480 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.069980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.069626 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.069980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.069723 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.070152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.070066 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.070215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.070180 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.070267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.070215 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.070507 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.070468 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.070639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.070532 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.070639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.070575 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjxd\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-kube-api-access-cpjxd\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.070639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.070616 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.070801 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.070776 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.071212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.071173 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.071382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.071248 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.074738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.074654 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.075146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.075108 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.093806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.093710 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.093806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.093764 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjxd\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-kube-api-access-cpjxd\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.205505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.205417 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:39.847381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:39.847277 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:40.222868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:40.214919 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9"] Oct 30 18:15:40.273614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:40.263304 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:40.309038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:40.280708 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:40.309038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:40.280953 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:40.309038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:40.295060 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:41.492987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:41.351028 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:15:41.492987 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:41.426740 5624 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6dbd6a2_df14_4e02_8e23_8ff769a8d6f3.slice/crio-542dc67a2e64927e5d8153fa9809469740c492de80d3e1756e9586cded3cc97f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:15:41.492987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:41.464907 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:41.864334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:41.864227 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.119129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.114298 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:42.197842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.184092 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podStartSLOduration=4.184051987 podStartE2EDuration="4.184051987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:42.067420185 +0000 UTC m=+836.455948304" watchObservedRunningTime="2026-10-30 18:15:42.184051987 +0000 UTC m=+836.572580105" Oct 30 18:15:42.247977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.233618 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" event={"ID":"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3","Type":"ContainerStarted","Data":"542dc67a2e64927e5d8153fa9809469740c492de80d3e1756e9586cded3cc97f"} Oct 30 18:15:42.278819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.278723 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" event={"ID":"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3","Type":"ContainerStarted","Data":"cc0616140386cceb1fde1ed50db245d5f4b8f5cc1c31f23b9d50bfcd7937752f"} Oct 30 18:15:42.280193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.280130 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:42.297981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.240639 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:42.298508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.298423 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:42.349564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.347172 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.435986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.434971 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.436409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.436163 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:15:42.574245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.574122 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.899087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.867061 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:15:42.899087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.868321 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:15:42.978997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.974324 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:15:43.012988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:42.977461 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:43.390161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.348108 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:43.456268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.456166 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:43.554206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.482215 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:15:43.590315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.590228 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:15:43.876082 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.867584 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:44.194046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.563583 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:15:44.201201 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.886517 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:44.321516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.973926 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:44.322288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.322203 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:44.329435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.323329 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:44.330076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.330008 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:44.330395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.129428 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:15:44.351213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.993020 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:15:44.430093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.351107 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:15:44.452543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:43.715604 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:44.481522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.020759 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:44.482258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.482221 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:44.482532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.482486 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:15:44.531995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.262223 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 18:15:44.712196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.378514 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:44.712196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.712154 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:44.769996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.766080 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:44.840303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:44.816434 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:45.652925 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:45.250575 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:45.663793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:45.341132 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:45.664745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:45.341261 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:46.027099 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:45.341310 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:15:46.414330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:45.665057 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:15:46.414330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:45.727531 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:46.414330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.336086 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:46.419021 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:46.416775 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.934s" Oct 30 18:15:46.662949 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.121205 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:46.662949 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.132310 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:46.662949 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.132683 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded" Oct 30 18:15:46.662949 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:45.730451 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 18:15:46.662949 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:45.730537 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:46.787003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.773164 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:15:46.787003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.773801 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:46.787003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.776979 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:46.802122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.788602 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:46.802122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.788805 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:15:46.802596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.802145 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:46.809117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.809018 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:46.874438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.848203 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:47.078551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.811238 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:47.079711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.811736 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:47.109858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.109726 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:47.130118 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:46.940221 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:47.131160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.054164 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:47.131835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.131664 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:47.155224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.143537 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:47.318372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.318264 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:47.334517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.334271 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:15:47.608185 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.591576 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:47.608185 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.591693 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:47.631855 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:47.631758 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.02s" Oct 30 18:15:47.869265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.706461 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Oct 30 18:15:47.932080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:47.716377 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:48.181051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.180995 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:15:48.187191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.187113 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:15:48.187815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.187776 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:15:48.207641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.207564 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:48.218471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.218088 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:48.423486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.420160 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:15:48.423486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.421379 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:15:48.544124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.534333 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:15:48.854885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.562057 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Oct 30 18:15:48.854885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.562116 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:15:48.854885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.562147 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:15:48.854885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.853202 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:15:48.854885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.708919 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:15:48.854885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.562194 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:15:48.854885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.709016 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 18:15:48.980315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:48.980269 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:49.072958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.069086 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:15:49.072958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.070371 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:15:49.072958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.070423 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:15:49.081617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.076866 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:15:49.087147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.084819 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:15:49.135305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.135259 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:15:49.138278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.138223 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:49.144383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.144324 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:15:49.243850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.243494 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:49.243850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:49.243618 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:49.973587 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:15:49.973510 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:15:50.208894 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:50.208800 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:50.209500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:50.209437 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:50.727595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:50.727509 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:51.209978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:51.209775 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:51.210389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:51.209912 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:52.206442 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:52.206403 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:52.207103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:52.207071 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:52.365048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:52.364998 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:53.219678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.219634 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:53.220480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.220439 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:53.706527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.706470 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq"] Oct 30 18:15:53.708496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.708435 5624 cpu_manager.go:414] "containerMap: removing container" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:53.708871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.708845 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:53.709103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.709086 5624 cpu_manager.go:414] "containerMap: removing container" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:53.709203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.709191 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:53.709989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.709949 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3abc77b-667b-4af2-bc8a-9f0e4b00889d" containerName="cleanup" Oct 30 18:15:53.710237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.710221 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddefc02-4b82-4f04-ad2b-a8f42d76724e" containerName="cleanup" Oct 30 18:15:53.710360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.710348 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="e836cff8-aa85-4cb1-ac26-36b0874bba78" containerName="cleanup" Oct 30 18:15:53.719418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.718873 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.741766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.741708 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:15:53.742162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.741805 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:15:53.742162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.741856 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:15:53.742162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.742098 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxwr\"" Oct 30 18:15:53.742162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.742143 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq"] Oct 30 18:15:53.861099 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.861020 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghxc\" (UniqueName: \"kubernetes.io/projected/9249b0ef-7632-4d51-9254-5e0e95c82313-kube-api-access-mghxc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.861446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.861288 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.861600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.861565 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.861736 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.861708 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.861853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.861828 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.862105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.862074 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.862288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.862135 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9249b0ef-7632-4d51-9254-5e0e95c82313-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.964533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.964363 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.964533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.964447 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.964533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.964532 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.964921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.964635 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.964921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.964665 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9249b0ef-7632-4d51-9254-5e0e95c82313-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.965130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.965067 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghxc\" (UniqueName: \"kubernetes.io/projected/9249b0ef-7632-4d51-9254-5e0e95c82313-kube-api-access-mghxc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.965603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.965244 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.965877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.965798 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.968283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.968140 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9249b0ef-7632-4d51-9254-5e0e95c82313-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.968598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.968354 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.968598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.968404 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.968598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.968354 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.987168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.987125 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9249b0ef-7632-4d51-9254-5e0e95c82313-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:53.987585 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:53.987542 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghxc\" (UniqueName: \"kubernetes.io/projected/9249b0ef-7632-4d51-9254-5e0e95c82313-kube-api-access-mghxc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4drwq\" (UID: \"9249b0ef-7632-4d51-9254-5e0e95c82313\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:54.027724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:54.027655 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:54.058039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:54.057924 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:54.218614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:54.218132 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:54.230146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:54.230079 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:15:54.231283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:54.231210 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:15:54.579679 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:54.579590 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq"] Oct 30 18:15:54.582045 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:15:54.581993 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9249b0ef_7632_4d51_9254_5e0e95c82313.slice/crio-b360abc5befa1c733776116bfbd96b566d8f0a43e329bfb60e01e8976ba44e70 WatchSource:0}: Error finding container b360abc5befa1c733776116bfbd96b566d8f0a43e329bfb60e01e8976ba44e70: Status 404 returned error can't find the container with id b360abc5befa1c733776116bfbd96b566d8f0a43e329bfb60e01e8976ba44e70 Oct 30 18:15:54.587186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:54.587117 5624 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:15:55.216118 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:55.216062 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:55.218869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:55.218799 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:15:55.260283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:55.260247 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" event={"ID":"9249b0ef-7632-4d51-9254-5e0e95c82313","Type":"ContainerStarted","Data":"b360abc5befa1c733776116bfbd96b566d8f0a43e329bfb60e01e8976ba44e70"} Oct 30 18:15:55.731154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:55.731116 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:57.332564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:57.332488 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:58.559169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:58.559084 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:58.559837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:58.559236 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:15:59.030637 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:59.030571 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:59.319066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:59.318926 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" event={"ID":"9249b0ef-7632-4d51-9254-5e0e95c82313","Type":"ContainerStarted","Data":"a891e5089adb2119afbb945f4ed404f8d472d6fbbd7c9814464d15116ba8d6c5"} Oct 30 18:15:59.319066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:59.319025 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:15:59.321865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:59.321818 5624 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-4drwq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:15:59.322103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:59.321888 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" podUID="9249b0ef-7632-4d51-9254-5e0e95c82313" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:15:59.378328 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:15:59.378189 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" podStartSLOduration=2.4093710919999998 podStartE2EDuration="6.378158015s" totalImagesPullingTime="3.968786923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:59.374352361 +0000 UTC m=+853.762880491" watchObservedRunningTime="2026-10-30 18:15:59.378158015 +0000 UTC m=+853.766686132" Oct 30 18:16:00.333264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:00.333193 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4drwq" Oct 30 18:16:00.608663 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:00.608480 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:02.122408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:02.122340 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:03.720349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:03.720286 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:05.436734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:05.436656 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:06.946645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:06.946563 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:07.267244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:07.267072 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:08.876960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:08.876016 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:16:08.921956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:08.921719 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr"] Oct 30 18:16:09.005061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.004136 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr"] Oct 30 18:16:09.005061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.004273 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116063 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116127 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023e0604-0666-4759-a030-cf93f46ebb78-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116154 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116198 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116245 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023e0604-0666-4759-a030-cf93f46ebb78-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116295 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116503 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116554 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116685 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dkj4\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-kube-api-access-6dkj4\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.116843 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.184133 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.184419 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:16:09.205183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.186022 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219242 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219312 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219434 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dkj4\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-kube-api-access-6dkj4\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219492 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219524 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023e0604-0666-4759-a030-cf93f46ebb78-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219549 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219588 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219635 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023e0604-0666-4759-a030-cf93f46ebb78-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.219682 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.220368 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.220618 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.264401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.220845 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.470095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.304187 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023e0604-0666-4759-a030-cf93f46ebb78-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.470095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.331492 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.470095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.363587 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023e0604-0666-4759-a030-cf93f46ebb78-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.470095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.435573 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.492961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.476820 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.492961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.489251 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.661979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.630432 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.661979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.634778 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dkj4\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-kube-api-access-6dkj4\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:09.942095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.861796 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:10.035182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.886325 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:10.035182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.921393 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:10.035182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:09.972521 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:10.402273 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:10.344184 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:16:10.516482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:10.516435 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:10.668097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:10.667983 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:16:10.668557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:10.668535 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:16:10.668827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:10.668800 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:16:10.669284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:10.669265 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:16:10.858225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:10.858139 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:16:10.888175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:10.887897 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:16:11.355897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.355685 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr"] Oct 30 18:16:11.362878 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:16:11.362793 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023e0604_0666_4759_a030_cf93f46ebb78.slice/crio-1927baeed3fa91cf612ebede25614778dad743d8e25518e4cb31d9eefc2e573c WatchSource:0}: Error finding container 1927baeed3fa91cf612ebede25614778dad743d8e25518e4cb31d9eefc2e573c: Status 404 returned error can't find the container with id 1927baeed3fa91cf612ebede25614778dad743d8e25518e4cb31d9eefc2e573c Oct 30 18:16:11.368897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.368806 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:16:11.369282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.368976 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:16:11.369282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.369027 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:16:11.897980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.895132 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" event={"ID":"023e0604-0666-4759-a030-cf93f46ebb78","Type":"ContainerStarted","Data":"d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4"} Oct 30 18:16:11.897980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.895382 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" event={"ID":"023e0604-0666-4759-a030-cf93f46ebb78","Type":"ContainerStarted","Data":"1927baeed3fa91cf612ebede25614778dad743d8e25518e4cb31d9eefc2e573c"} Oct 30 18:16:11.929078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.928853 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:11.955469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.955338 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podStartSLOduration=3.95530156 podStartE2EDuration="3.95530156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:11.950059464 +0000 UTC m=+866.338587597" watchObservedRunningTime="2026-10-30 18:16:11.95530156 +0000 UTC m=+866.343829677" Oct 30 18:16:11.978730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:11.978653 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:12.931580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:12.931466 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.932430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:12.931587 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 18:16:13.929307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:13.929249 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.930116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:13.930064 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 18:16:14.344872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:14.344784 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:14.928038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:14.927982 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.928321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:14.928069 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 18:16:15.928288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:15.928243 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.928776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:15.928308 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 18:16:16.449087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:16.447003 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:16.928437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:16.928408 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.928924 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:16.928901 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 18:16:17.056264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:17.056225 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:17.932108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:17.930439 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:16:17.932108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:17.930574 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 18:16:18.461879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:18.461807 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:18.931845 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:18.931754 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:16:18.932530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:18.932470 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 18:16:19.928590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:19.928549 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 18:16:19.928775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:19.928614 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 18:16:19.929868 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:16:19.929837 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:16:20.460811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:20.460771 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:20.934512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:20.934472 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:20.935016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:20.934928 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:20.936381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:20.936348 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:16:22.113588 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:22.113559 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:23.790336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:23.790288 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:25.241653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:25.241604 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:26.832148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:26.832064 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:28.416894 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:28.416839 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:30.045739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:30.045708 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:31.639371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:31.639301 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:33.285191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:33.285121 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:33.602121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:33.601913 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:16:35.570960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:35.568059 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:36.492110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:36.492061 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.776781 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:36.776686 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:16:36.829078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:36.829025 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.896126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:36.896084 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.106149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:37.106025 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.153072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:37.153023 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.162690 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:37.162656 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.330956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:37.330655 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:16:37.516957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:37.516700 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.533952 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:37.517159 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:16:37.969809 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:37.969755 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.999101 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:37.999053 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.007782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.006185 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:16:38.032960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.016292 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.060390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.037590 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:16:38.116959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.091299 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.169957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.167546 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.362065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.353712 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.362065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.354114 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 18:16:38.545022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.519576 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.954961 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:16:38.945508 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.161s" Oct 30 18:16:38.954961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.945593 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:16:38.954961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.945718 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:16:38.954961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.945757 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:16:38.954961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.945792 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:16:38.954961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.945828 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:16:38.999958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:38.998027 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.064259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.064200 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.176895 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.168114 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:16:39.176895 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.168577 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:16:39.206293 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.194204 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.221959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.210799 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:16:39.223458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.223421 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:16:39.352957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.351342 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz"] Oct 30 18:16:39.361248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.361196 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.399996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.377171 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.591023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.540812 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.551039 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.552838 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.627194 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.627240 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.627334 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.627394 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddccd18-fc91-4f95-8783-5042793d08ac-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.627430 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.627500 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.627615 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxvtz\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-kube-api-access-xxvtz\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.627687 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.642968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.640386 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz"] Oct 30 18:16:39.656973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.656918 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:16:39.781339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.732719 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.793089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.789084 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxvtz\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-kube-api-access-xxvtz\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.836959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.808385 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.925086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.912380 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.925086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.912563 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.925086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.912607 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.925086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.912711 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.925086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.912782 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddccd18-fc91-4f95-8783-5042793d08ac-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.925086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.912823 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.941009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.926128 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.941009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.839948 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.941009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.926422 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.941009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.926654 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.966335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.932008 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddccd18-fc91-4f95-8783-5042793d08ac-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:39.966335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:39.941628 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:40.024956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.012240 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:16:40.133769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.132601 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:40.189083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.184006 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.215958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.204051 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:40.251284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.248673 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:16:40.278374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.248726 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:16:40.339810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.339775 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxvtz\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-kube-api-access-xxvtz\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:40.446321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.296951 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:40.467999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.450828 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:16:40.547997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.539983 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:40.596111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.594725 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:40.661308 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.661252 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.703041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:40.677657 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:16:41.109422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.029995 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.144959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.135855 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.228137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.228093 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.287438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.283970 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.367010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.349241 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:16:41.367010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.350855 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.458990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.458094 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.607526 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.564057 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:41.607526 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.564124 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.653484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.653454 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:16:41.653756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.653737 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:16:41.679013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.675751 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.810732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.810649 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:16:41.865827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.865799 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:16:41.866069 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.865852 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:16:41.866069 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.865880 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:16:41.895839 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.895807 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:16:41.906664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.905079 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:16:41.996468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:41.996438 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz"] Oct 30 18:16:42.004241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:42.004205 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:16:42.004434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:42.004414 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:16:42.004567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:42.004550 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:16:42.150585 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:42.150547 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" event={"ID":"0ddccd18-fc91-4f95-8783-5042793d08ac","Type":"ContainerStarted","Data":"78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608"} Oct 30 18:16:42.151070 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:42.150615 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" event={"ID":"0ddccd18-fc91-4f95-8783-5042793d08ac","Type":"ContainerStarted","Data":"094ab4d43b302cf7693e891c28aa866c2019fd016ce298757cf8efe63220eb5b"} Oct 30 18:16:42.230880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:42.230762 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podStartSLOduration=3.230742667 podStartE2EDuration="3.230742667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:42.229265493 +0000 UTC m=+896.617793613" watchObservedRunningTime="2026-10-30 18:16:42.230742667 +0000 UTC m=+896.619270784" Oct 30 18:16:42.608011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:42.598034 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:42.659320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:42.659220 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:43.611093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:43.610048 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:43.611093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:43.610111 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:44.600674 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:44.598062 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:44.606437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:44.604267 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:44.659857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:44.659790 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:45.624345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:45.622484 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:45.624345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:45.622565 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:46.568219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:46.568183 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:46.604011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:46.601339 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:46.604011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:46.601459 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:47.607985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:47.598894 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:47.607985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:47.600888 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:47.720637 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:47.720589 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:48.529473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:48.529423 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:48.597316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:48.597280 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:16:48.598029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:48.597996 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:16:49.848022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:49.847992 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:49.849444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:49.849419 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:49.850772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:49.850755 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:16:49.923879 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:16:49.923840 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:16:50.321247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:50.321215 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:52.141349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:52.141311 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:53.797507 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:53.797363 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:55.442549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:55.442515 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:58.186311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:58.139118 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:16:58.408959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:58.391035 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:16:58.552491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:58.551047 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.709085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:58.709039 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.932413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:58.932364 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.200141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.200033 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.228014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.215297 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.228014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.215402 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:16:59.228014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.217819 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:16:59.391015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.342880 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:16:59.639023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.391319 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.639023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.487649 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:59.639023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.554451 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.654033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.653999 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:16:59.674096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.674061 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:16:59.727956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.722516 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:16:59.780096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.780063 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:16:59.986011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:16:59.984920 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.214956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.210210 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.267002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.216636 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.312961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.293599 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:17:00.314233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.314207 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:17:00.427960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.422010 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:17:00.599014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.580579 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:00.621682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.621647 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:17:00.622986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.622955 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.668968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.623648 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.763337 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.763284 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.915736 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.915698 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:00.997084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:00.997033 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:17:01.048878 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.048838 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:01.241724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.236536 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:17:01.252960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.243762 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.291960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.286256 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 30 18:17:01.431972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.425010 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.449967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.445392 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:17:01.496025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.495343 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:01.496025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.495404 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.583745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.583700 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:01.648084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.648033 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.667350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.648348 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:17:01.816093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:01.789904 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.004961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:02.004319 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.286810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:02.286756 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.342958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:02.338914 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:17:02.580054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:02.554875 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.842017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:02.805075 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:03.022960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:03.022635 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:03.022960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:03.022733 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.022960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:02.830371 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:17:03.109522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:02.830248 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:03.277051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:03.258628 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:17:03.291292 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:03.291249 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.580072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:03.580009 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:17:03.617152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:03.617092 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:17:03.633967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:03.628828 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:17:03.922982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:03.884028 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.028959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.024020 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.093959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.090565 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.280008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.274274 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.639469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.637296 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:17:04.650978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.639990 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:17:04.670964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.670270 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.822628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.822572 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:04.830332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.830304 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:17:04.832028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.831984 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.841097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.840514 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:17:04.869600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.844443 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:17:04.869600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.863163 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:17:04.870404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.870376 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:17:04.872307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.872068 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:17:04.883354 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.883326 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:17:04.883484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:04.883469 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:17:06.471684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:06.471650 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:07.179650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.179615 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm"] Oct 30 18:17:07.184466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.184444 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.203278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.203252 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm"] Oct 30 18:17:07.209211 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.209188 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:07.209348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.209309 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:17:07.375094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.375062 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.375314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.375293 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8n29\" (UniqueName: \"kubernetes.io/projected/1ab5896a-19cb-4175-a41a-3f1cc183990c-kube-api-access-x8n29\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.375420 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.375405 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.477158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.477077 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.477672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.477240 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8n29\" (UniqueName: \"kubernetes.io/projected/1ab5896a-19cb-4175-a41a-3f1cc183990c-kube-api-access-x8n29\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.477672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.477312 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.477930 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.477911 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.479554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.479531 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.497493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.497459 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8n29\" (UniqueName: \"kubernetes.io/projected/1ab5896a-19cb-4175-a41a-3f1cc183990c-kube-api-access-x8n29\") pod \"maas-api-revoke-sub-f37fa868-g5zcm\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.502202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.502178 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:07.811590 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:17:07.811563 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab5896a_19cb_4175_a41a_3f1cc183990c.slice/crio-093f06cc6b012fa5b573b64a606fd24374d0323e8011c8ec1726c049ff23ec53 WatchSource:0}: Error finding container 093f06cc6b012fa5b573b64a606fd24374d0323e8011c8ec1726c049ff23ec53: Status 404 returned error can't find the container with id 093f06cc6b012fa5b573b64a606fd24374d0323e8011c8ec1726c049ff23ec53 Oct 30 18:17:07.814554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.814526 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm"] Oct 30 18:17:07.932875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:07.932832 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" event={"ID":"1ab5896a-19cb-4175-a41a-3f1cc183990c","Type":"ContainerStarted","Data":"093f06cc6b012fa5b573b64a606fd24374d0323e8011c8ec1726c049ff23ec53"} Oct 30 18:17:08.179005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:08.178929 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:08.946464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:08.946427 5624 generic.go:356] "Generic (PLEG): container finished" podID="1ab5896a-19cb-4175-a41a-3f1cc183990c" containerID="14a5a749352d09cfa86aa375fc99787efe901ec7978e8450c0c414be183cc511" exitCode=0 Oct 30 18:17:08.946963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:08.946495 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" event={"ID":"1ab5896a-19cb-4175-a41a-3f1cc183990c","Type":"ContainerDied","Data":"14a5a749352d09cfa86aa375fc99787efe901ec7978e8450c0c414be183cc511"} Oct 30 18:17:09.082819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.082746 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65"] Oct 30 18:17:09.087728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.087705 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.109420 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.109619 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.110148 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.110222 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqf6g\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-kube-api-access-vqf6g\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.110339 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.110408 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.110496 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0c98c9-ca4c-430f-a465-edc7b68da522-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.110568 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.110967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.110655 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.124144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.124057 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65"] Oct 30 18:17:09.212161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212117 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212178 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqf6g\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-kube-api-access-vqf6g\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212233 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212273 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212316 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0c98c9-ca4c-430f-a465-edc7b68da522-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212362 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212409 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212474 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212508 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212814 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212630 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.212814 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212758 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.213012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.212992 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.213142 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.213122 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.213243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.213210 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0c98c9-ca4c-430f-a465-edc7b68da522-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.214874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.214851 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.215234 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.215210 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.232898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.232874 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.233340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.233319 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqf6g\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-kube-api-access-vqf6g\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.424867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.424832 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:09.755593 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:17:09.755564 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0c98c9_ca4c_430f_a465_edc7b68da522.slice/crio-089b31fd5e5b60e81a8cf8f038eb0149911bb36b16a0f87092868294791fbb9d WatchSource:0}: Error finding container 089b31fd5e5b60e81a8cf8f038eb0149911bb36b16a0f87092868294791fbb9d: Status 404 returned error can't find the container with id 089b31fd5e5b60e81a8cf8f038eb0149911bb36b16a0f87092868294791fbb9d Oct 30 18:17:09.756042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.756021 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65"] Oct 30 18:17:09.758826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.758791 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:17:09.758969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.758869 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:17:09.758969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.758917 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:17:09.934692 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.934652 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:09.962624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.962587 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" event={"ID":"1d0c98c9-ca4c-430f-a465-edc7b68da522","Type":"ContainerStarted","Data":"850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273"} Oct 30 18:17:09.963139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:09.962647 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" event={"ID":"1d0c98c9-ca4c-430f-a465-edc7b68da522","Type":"ContainerStarted","Data":"089b31fd5e5b60e81a8cf8f038eb0149911bb36b16a0f87092868294791fbb9d"} Oct 30 18:17:10.020036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.017688 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podStartSLOduration=1.017666835 podStartE2EDuration="1.017666835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:10.010310415 +0000 UTC m=+924.398838536" watchObservedRunningTime="2026-10-30 18:17:10.017666835 +0000 UTC m=+924.406194956" Oct 30 18:17:10.294210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.294147 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:10.350871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.350826 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-openshift-ca\") pod \"1ab5896a-19cb-4175-a41a-3f1cc183990c\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " Oct 30 18:17:10.351118 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.351030 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-xks-ca\") pod \"1ab5896a-19cb-4175-a41a-3f1cc183990c\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " Oct 30 18:17:10.351225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.351193 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab5896a-19cb-4175-a41a-3f1cc183990c-kube-api-access-x8n29\" (UniqueName: \"kubernetes.io/projected/1ab5896a-19cb-4175-a41a-3f1cc183990c-kube-api-access-x8n29\") pod \"1ab5896a-19cb-4175-a41a-3f1cc183990c\" (UID: \"1ab5896a-19cb-4175-a41a-3f1cc183990c\") " Oct 30 18:17:10.351309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.351271 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-openshift-ca" pod "1ab5896a-19cb-4175-a41a-3f1cc183990c" (UID: "1ab5896a-19cb-4175-a41a-3f1cc183990c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:10.351910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.351860 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.354342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.354314 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab5896a-19cb-4175-a41a-3f1cc183990c-kube-api-access-x8n29" pod "1ab5896a-19cb-4175-a41a-3f1cc183990c" (UID: "1ab5896a-19cb-4175-a41a-3f1cc183990c"). InnerVolumeSpecName "kube-api-access-x8n29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:10.354560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.354540 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-xks-ca" pod "1ab5896a-19cb-4175-a41a-3f1cc183990c" (UID: "1ab5896a-19cb-4175-a41a-3f1cc183990c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:10.425615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.425586 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:10.427107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.427081 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:17:10.427245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.427126 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:17:10.455391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.455326 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab5896a-19cb-4175-a41a-3f1cc183990c-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.455391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.455363 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8n29\" (UniqueName: \"kubernetes.io/projected/1ab5896a-19cb-4175-a41a-3f1cc183990c-kube-api-access-x8n29\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.971492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.971464 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" Oct 30 18:17:10.971492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.971470 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm" event={"ID":"1ab5896a-19cb-4175-a41a-3f1cc183990c","Type":"ContainerDied","Data":"093f06cc6b012fa5b573b64a606fd24374d0323e8011c8ec1726c049ff23ec53"} Oct 30 18:17:10.972060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:10.971523 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="093f06cc6b012fa5b573b64a606fd24374d0323e8011c8ec1726c049ff23ec53" Oct 30 18:17:11.425650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:11.425618 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:17:11.425862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:11.425678 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:17:11.745102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:11.745023 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:12.427043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:12.425852 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:17:12.427043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:12.425900 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:17:13.425995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:13.425928 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:17:13.426180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:13.426022 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:17:13.540737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:13.540684 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:14.425878 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:14.425838 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:17:14.426096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:14.425898 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:17:15.244404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:15.244372 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:15.426218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:15.426186 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:17:15.426504 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:15.426464 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:17:16.425825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:16.425787 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:17:16.426366 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:16.425863 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 18:17:16.981651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:16.981618 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:17.508267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:17.508233 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:17.509677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:17.509652 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:17.510790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:17.510766 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:17:18.720690 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:18.720650 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:19.918872 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:17:19.918833 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:17:20.433247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:20.433212 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:22.137520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:22.137490 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:23.218973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:23.218656 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:17:23.827304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:23.827262 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:26.011155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:26.011097 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.599062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:26.599017 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.633979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:26.622421 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.633979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:26.629152 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:26.780403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:26.764228 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:17:26.865148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:26.865045 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:17:27.054978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.035056 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.054978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.035498 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:17:27.138482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.138442 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.244081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.244039 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.244681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.244651 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Oct 30 18:17:27.436018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.370040 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:17:27.648970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.600978 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:27.648970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.601520 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:17:27.712014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.687834 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:17:27.763402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.700003 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:27.862854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.767195 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:17:27.929557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.767530 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:17:28.056777 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:27.888483 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.061007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.049139 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.109085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.103606 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:17:28.136835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.136809 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:17:28.137107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.137091 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:17:28.147059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.110814 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.303347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.299025 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:17:28.303347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.299567 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:17:28.412072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.412029 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.643837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.461379 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.645652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.645626 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:17:28.669542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.647001 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.774086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:28.773415 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.224724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.210469 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.313731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.246903 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 18:17:29.313731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.258110 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.313731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.269045 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:17:29.313731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.271328 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.465968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.447864 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:17:29.656984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.656799 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:17:29.718972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.697411 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:29.836501 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:29.836459 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.239194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:30.239148 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.285710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:30.271221 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.285710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:30.285302 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:17:30.345175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:30.319712 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.464144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:30.464091 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:17:30.661594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:30.661553 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.700447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:30.700401 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.907959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:30.894492 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:17:31.060015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.057035 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:31.134958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.123265 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:17:31.259651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.252863 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:17:31.259651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.240588 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:31.277134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.276822 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:31.285268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.283661 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:17:31.285831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.285812 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:17:31.286947 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.286915 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:17:31.287109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.287097 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:17:31.300223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.300204 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:17:31.302648 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:31.302630 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:17:33.541645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:33.541613 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:35.256007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:35.255966 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:37.082576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:37.082544 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:38.839434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:38.839401 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:40.806705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:40.806671 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:41.225898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:41.225844 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:42.470961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:42.463870 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.940800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:42.940754 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:42.948958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:42.945464 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:17:42.955036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:42.954996 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:17:43.291016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.278732 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.291016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.279623 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:17:43.291016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.286273 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:43.462959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.460070 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.484971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.481514 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.484971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.483187 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:17:43.963956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.963246 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.968958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.967208 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.990958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:43.972309 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.185965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.179693 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:17:44.185965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.179794 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:17:44.185965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.179855 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:17:44.339958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.339822 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:17:44.354958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.342127 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:17:44.504561 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.504515 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.563119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.563079 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:17:44.564061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.564030 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:44.722960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.697419 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:17:44.900249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.900223 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:17:44.900597 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.900578 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:17:45.006037 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.004318 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.007084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:44.988498 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:45.032796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.027023 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.483017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.396202 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:17:45.486805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.486766 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:17:45.560872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.560831 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.567259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.567228 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:17:45.569353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.569327 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:17:45.729789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.729760 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:17:45.732480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.732459 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:17:45.913575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:45.913531 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded" Oct 30 18:17:46.024972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.024253 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:46.057536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.057498 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:46.057827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.057804 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:17:46.063965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.062740 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:17:46.063965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.062797 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:17:46.067045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.067028 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:17:46.133613 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.133585 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:17:46.157875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.144537 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:17:46.157875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.155275 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:17:46.883073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:46.882901 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.844572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:47.844488 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.942628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:47.942548 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:48.545889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:48.545857 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:48.838060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:48.837895 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:49.872064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.871878 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:49.950402 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:17:49.950339 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:17:49.953982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.953889 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v"] Oct 30 18:17:49.955825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.955748 5624 cpu_manager.go:414] "containerMap: removing container" podUID="1ab5896a-19cb-4175-a41a-3f1cc183990c" containerName="revoke-keys" Oct 30 18:17:49.955825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.955827 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab5896a-19cb-4175-a41a-3f1cc183990c" containerName="revoke-keys" Oct 30 18:17:49.956305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.956275 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab5896a-19cb-4175-a41a-3f1cc183990c" containerName="revoke-keys" Oct 30 18:17:49.963567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.963511 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:49.979746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.979534 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:49.983124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.983085 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v"] Oct 30 18:17:49.987279 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.987213 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:49.987736 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:49.987226 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:17:50.065520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.065440 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhnm\" (UniqueName: \"kubernetes.io/projected/4abc93f2-30f0-4fd9-bfd2-262775c596a3-kube-api-access-khhnm\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.066246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.066201 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.066483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.066398 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.169582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.169400 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.169881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.169651 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhnm\" (UniqueName: \"kubernetes.io/projected/4abc93f2-30f0-4fd9-bfd2-262775c596a3-kube-api-access-khhnm\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.170116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.170088 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.171202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.171161 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.173705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.173647 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.174315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.174263 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:50.192770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.192585 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhnm\" (UniqueName: \"kubernetes.io/projected/4abc93f2-30f0-4fd9-bfd2-262775c596a3-kube-api-access-khhnm\") pod \"maas-api-revoke-sub-22dc722a-cbj4v\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.299513 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.299479 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:50.550883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.550791 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:50.615532 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:17:50.615484 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abc93f2_30f0_4fd9_bfd2_262775c596a3.slice/crio-1d03749abb0b9526294952f4871c673d21665405c75b9d3d53ab2b372d117718 WatchSource:0}: Error finding container 1d03749abb0b9526294952f4871c673d21665405c75b9d3d53ab2b372d117718: Status 404 returned error can't find the container with id 1d03749abb0b9526294952f4871c673d21665405c75b9d3d53ab2b372d117718 Oct 30 18:17:50.617082 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.617048 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v"] Oct 30 18:17:50.649426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.649395 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:50.748542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:50.748504 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:51.417544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:51.417459 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" event={"ID":"4abc93f2-30f0-4fd9-bfd2-262775c596a3","Type":"ContainerStarted","Data":"6a4034dc94910b6e8238b469dfda6c23e982a028f1aa00a2f87808d961d20f58"} Oct 30 18:17:51.418452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:51.417586 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" event={"ID":"4abc93f2-30f0-4fd9-bfd2-262775c596a3","Type":"ContainerStarted","Data":"1d03749abb0b9526294952f4871c673d21665405c75b9d3d53ab2b372d117718"} Oct 30 18:17:51.459732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:51.459601 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" podStartSLOduration=2.459559635 podStartE2EDuration="2.459559635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:51.457890596 +0000 UTC m=+965.846418714" watchObservedRunningTime="2026-10-30 18:17:51.459559635 +0000 UTC m=+965.848087752" Oct 30 18:17:52.504370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:52.504283 5624 generic.go:356] "Generic (PLEG): container finished" podID="4abc93f2-30f0-4fd9-bfd2-262775c596a3" containerID="6a4034dc94910b6e8238b469dfda6c23e982a028f1aa00a2f87808d961d20f58" exitCode=0 Oct 30 18:17:52.505308 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:52.504655 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" event={"ID":"4abc93f2-30f0-4fd9-bfd2-262775c596a3","Type":"ContainerDied","Data":"6a4034dc94910b6e8238b469dfda6c23e982a028f1aa00a2f87808d961d20f58"} Oct 30 18:17:52.558170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:52.558100 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:53.740146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.740104 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:53.833526 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.833411 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-xks-ca\") pod \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " Oct 30 18:17:53.835320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.835286 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-openshift-ca\") pod \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " Oct 30 18:17:53.836173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.836139 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-openshift-ca" pod "4abc93f2-30f0-4fd9-bfd2-262775c596a3" (UID: "4abc93f2-30f0-4fd9-bfd2-262775c596a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:53.836554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.836532 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abc93f2-30f0-4fd9-bfd2-262775c596a3-kube-api-access-khhnm\" (UniqueName: \"kubernetes.io/projected/4abc93f2-30f0-4fd9-bfd2-262775c596a3-kube-api-access-khhnm\") pod \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\" (UID: \"4abc93f2-30f0-4fd9-bfd2-262775c596a3\") " Oct 30 18:17:53.837768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.837739 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:17:53.842703 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.842666 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-xks-ca" pod "4abc93f2-30f0-4fd9-bfd2-262775c596a3" (UID: "4abc93f2-30f0-4fd9-bfd2-262775c596a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:53.844130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.844092 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abc93f2-30f0-4fd9-bfd2-262775c596a3-kube-api-access-khhnm" pod "4abc93f2-30f0-4fd9-bfd2-262775c596a3" (UID: "4abc93f2-30f0-4fd9-bfd2-262775c596a3"). InnerVolumeSpecName "kube-api-access-khhnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:53.939863 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.939820 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abc93f2-30f0-4fd9-bfd2-262775c596a3-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:17:53.939863 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:53.939858 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhnm\" (UniqueName: \"kubernetes.io/projected/4abc93f2-30f0-4fd9-bfd2-262775c596a3-kube-api-access-khhnm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:17:54.085283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:54.085091 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:54.193381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:54.193304 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:54.358234 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:54.358087 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:54.551976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:54.551629 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" event={"ID":"4abc93f2-30f0-4fd9-bfd2-262775c596a3","Type":"ContainerDied","Data":"1d03749abb0b9526294952f4871c673d21665405c75b9d3d53ab2b372d117718"} Oct 30 18:17:54.551976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:54.551765 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d03749abb0b9526294952f4871c673d21665405c75b9d3d53ab2b372d117718" Oct 30 18:17:54.551976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:54.551989 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v" Oct 30 18:17:54.819534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:54.819501 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:54.921175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:54.921144 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:55.063396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.063366 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:55.166109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.166056 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:55.239718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.239567 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w"] Oct 30 18:17:55.240873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.240837 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4abc93f2-30f0-4fd9-bfd2-262775c596a3" containerName="revoke-keys" Oct 30 18:17:55.240873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.240874 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abc93f2-30f0-4fd9-bfd2-262775c596a3" containerName="revoke-keys" Oct 30 18:17:55.241150 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.241109 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abc93f2-30f0-4fd9-bfd2-262775c596a3" containerName="revoke-keys" Oct 30 18:17:55.245760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.245709 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:55.266659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.266588 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w"] Oct 30 18:17:55.267111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.267079 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-535b743b\"" Oct 30 18:17:55.269815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.269771 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-535b743b-dockercfg-v2gjj\"" Oct 30 18:17:55.406819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.405353 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22277\" (UniqueName: \"kubernetes.io/projected/6d612bfa-5e32-4d59-8f30-40bef0c04c20-kube-api-access-22277\") pod \"payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w\" (UID: \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:55.406819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.406134 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d612bfa-5e32-4d59-8f30-40bef0c04c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w\" (UID: \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:55.756020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.753825 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d612bfa-5e32-4d59-8f30-40bef0c04c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w\" (UID: \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:55.756020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.754311 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22277\" (UniqueName: \"kubernetes.io/projected/6d612bfa-5e32-4d59-8f30-40bef0c04c20-kube-api-access-22277\") pod \"payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w\" (UID: \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:55.947962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.915220 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:56.001989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:55.987435 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d612bfa-5e32-4d59-8f30-40bef0c04c20-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w\" (UID: \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:56.245535 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.245498 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:56.563708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.550323 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr"] Oct 30 18:17:56.593518 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.593484 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22277\" (UniqueName: \"kubernetes.io/projected/6d612bfa-5e32-4d59-8f30-40bef0c04c20-kube-api-access-22277\") pod \"payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w\" (UID: \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:56.683013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.682788 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr"] Oct 30 18:17:56.683013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.682843 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p"] Oct 30 18:17:56.707323 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.707292 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:56.717214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.717184 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p"] Oct 30 18:17:56.717388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.717276 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.738103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.738077 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7cb9c50e-openshift-default-dockercfg-rvfbm\"" Oct 30 18:17:56.777213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.777185 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:56.798094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.798060 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6sv2\" (UniqueName: \"kubernetes.io/projected/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-kube-api-access-h6sv2\") pod \"payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr\" (UID: \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:56.798303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.798283 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr\" (UID: \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:56.835961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.835857 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:56.899737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.899698 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.901639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.899754 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23496535-1b6c-4a4d-8080-a06ce5a26b93-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.901639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.899809 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.901639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.899906 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.901639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.900056 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr\" (UID: \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:56.901639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.900086 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5wmp\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-kube-api-access-l5wmp\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.901639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.900168 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.901639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.900222 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.902076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.901825 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-plugins-config-volume\") pod \"payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr\" (UID: \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:56.902076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.900299 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.902076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.902002 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:56.902207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.902181 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6sv2\" (UniqueName: \"kubernetes.io/projected/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-kube-api-access-h6sv2\") pod \"payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr\" (UID: \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:56.925575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.925544 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6sv2\" (UniqueName: \"kubernetes.io/projected/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-kube-api-access-h6sv2\") pod \"payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr\" (UID: \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\") " pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:56.944151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:56.944122 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:57.020976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.014741 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5wmp\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-kube-api-access-l5wmp\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.020976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.014853 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.020976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.014898 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.020976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.015040 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.020976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.015306 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.039038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.015496 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.039216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.039056 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23496535-1b6c-4a4d-8080-a06ce5a26b93-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.039216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.039123 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.039216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.039207 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.039866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.039840 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.040076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.026804 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.053106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.040881 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23496535-1b6c-4a4d-8080-a06ce5a26b93-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.053402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.046442 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.053553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.046702 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.053681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.027341 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.053773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.037439 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.053885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.049367 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:57.070920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.058483 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.070920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.059589 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5wmp\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-kube-api-access-l5wmp\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.141833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.141362 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w"] Oct 30 18:17:57.347830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.347740 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:57.407098 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:17:57.407054 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30c309c_be3b_4a8a_83ed_8fcae7cb13bc.slice/crio-3d041e9051b03009657987044e903bcd1a3517ab78bce1474fb8b930ad30f96c WatchSource:0}: Error finding container 3d041e9051b03009657987044e903bcd1a3517ab78bce1474fb8b930ad30f96c: Status 404 returned error can't find the container with id 3d041e9051b03009657987044e903bcd1a3517ab78bce1474fb8b930ad30f96c Oct 30 18:17:57.413755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.412405 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr"] Oct 30 18:17:57.524268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.523981 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" event={"ID":"6d612bfa-5e32-4d59-8f30-40bef0c04c20","Type":"ContainerStarted","Data":"abeb34719bfc4ba4ef2e41545b95f557cbb33adc7586f05ece43bea1b45be307"} Oct 30 18:17:57.524463 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.524437 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:17:57.524553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.524479 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" event={"ID":"6d612bfa-5e32-4d59-8f30-40bef0c04c20","Type":"ContainerStarted","Data":"aa157d2f1fd9e3ea7c7c2560ac182135e1cf1785e9103e80efed789f4e3bb000"} Oct 30 18:17:57.526439 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.526404 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" event={"ID":"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc","Type":"ContainerStarted","Data":"d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328"} Oct 30 18:17:57.526542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.526459 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" event={"ID":"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc","Type":"ContainerStarted","Data":"3d041e9051b03009657987044e903bcd1a3517ab78bce1474fb8b930ad30f96c"} Oct 30 18:17:57.526747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.526726 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:17:57.565166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.565111 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" podStartSLOduration=2.56509298 podStartE2EDuration="2.56509298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:57.562737872 +0000 UTC m=+971.951265988" watchObservedRunningTime="2026-10-30 18:17:57.56509298 +0000 UTC m=+971.953621098" Oct 30 18:17:57.605440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.605338 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" podStartSLOduration=2.605324735 podStartE2EDuration="2.605324735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:57.601489776 +0000 UTC m=+971.990017894" watchObservedRunningTime="2026-10-30 18:17:57.605324735 +0000 UTC m=+971.993852853" Oct 30 18:17:57.667655 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:17:57.667618 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23496535_1b6c_4a4d_8080_a06ce5a26b93.slice/crio-f657e72bf50130c6a949b6847f04fc1629cd0245a41c5a5650a076bdf0b0d1b8 WatchSource:0}: Error finding container f657e72bf50130c6a949b6847f04fc1629cd0245a41c5a5650a076bdf0b0d1b8: Status 404 returned error can't find the container with id f657e72bf50130c6a949b6847f04fc1629cd0245a41c5a5650a076bdf0b0d1b8 Oct 30 18:17:57.669828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.669730 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p"] Oct 30 18:17:57.670987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.670908 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:17:57.671058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.671021 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:17:57.671102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:57.671065 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:17:58.538968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:58.538882 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" event={"ID":"23496535-1b6c-4a4d-8080-a06ce5a26b93","Type":"ContainerStarted","Data":"22ff70dbf386fbec9d2297213111f751f1b7facb13f22cd3cc460fb84b38dfac"} Oct 30 18:17:58.539433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:58.539102 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" event={"ID":"23496535-1b6c-4a4d-8080-a06ce5a26b93","Type":"ContainerStarted","Data":"f657e72bf50130c6a949b6847f04fc1629cd0245a41c5a5650a076bdf0b0d1b8"} Oct 30 18:17:58.602167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:58.602103 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podStartSLOduration=3.60208504 podStartE2EDuration="3.60208504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:58.601329832 +0000 UTC m=+972.989857951" watchObservedRunningTime="2026-10-30 18:17:58.60208504 +0000 UTC m=+972.990613158" Oct 30 18:17:58.634671 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:58.634640 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:59.348424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:59.348388 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:59.353743 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:59.353718 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:59.547350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:59.547319 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:17:59.548256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:17:59.548235 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:18:00.311614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:00.311576 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:00.414599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:00.414564 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:00.525962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:00.517284 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:01.282088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.282056 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:01.714448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.714416 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf"] Oct 30 18:18:01.730687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.719246 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.744378 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.743354 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f6a224e7-openshift-default-dockercfg-2ljbl\"" Oct 30 18:18:01.752275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.752237 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf"] Oct 30 18:18:01.762705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.762665 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867217 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647187d5-0217-4a90-a5c7-900553dfd802-istio-podinfo\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867310 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647187d5-0217-4a90-a5c7-900553dfd802-istiod-ca-cert\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867345 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-data\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867387 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-socket\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867431 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-certs\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867605 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-credential-socket\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867705 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-istio-token\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867817 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-envoy\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.884980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.867891 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2x76\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-kube-api-access-p2x76\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.912582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.912494 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972125 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2x76\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-kube-api-access-p2x76\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972236 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647187d5-0217-4a90-a5c7-900553dfd802-istio-podinfo\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972297 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647187d5-0217-4a90-a5c7-900553dfd802-istiod-ca-cert\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972330 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-data\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972371 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-socket\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972410 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-certs\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972547 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-credential-socket\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972644 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-istio-token\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.972757 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-envoy\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.973622 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-data\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.975022 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647187d5-0217-4a90-a5c7-900553dfd802-istiod-ca-cert\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.975269 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-certs\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.975515 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-socket\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.981000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.975759 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-credential-socket\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:01.985959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:01.982325 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-envoy\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:02.003738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.003661 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2x76\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-kube-api-access-p2x76\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:02.015731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.015699 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647187d5-0217-4a90-a5c7-900553dfd802-istio-podinfo\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:02.017650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.017619 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-istio-token\") pod \"e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:02.062958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.061898 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:02.246009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.239983 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:02.327958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.327540 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:02.439734 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:18:02.439689 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647187d5_0217_4a90_a5c7_900553dfd802.slice/crio-c8a36d25568b1e061ebd0066ce1996ee49be848ef68c4c54d83514afcf8bceb3 WatchSource:0}: Error finding container c8a36d25568b1e061ebd0066ce1996ee49be848ef68c4c54d83514afcf8bceb3: Status 404 returned error can't find the container with id c8a36d25568b1e061ebd0066ce1996ee49be848ef68c4c54d83514afcf8bceb3 Oct 30 18:18:02.441481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.441444 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf"] Oct 30 18:18:02.442843 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.442802 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:18:02.442987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.442880 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:18:02.442987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.442923 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:18:02.706967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.706374 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" event={"ID":"647187d5-0217-4a90-a5c7-900553dfd802","Type":"ContainerStarted","Data":"c8a36d25568b1e061ebd0066ce1996ee49be848ef68c4c54d83514afcf8bceb3"} Oct 30 18:18:02.913605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.913579 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:02.948014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:02.947730 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:18:03.845956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:03.828477 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" event={"ID":"647187d5-0217-4a90-a5c7-900553dfd802","Type":"ContainerStarted","Data":"f1276e6dbc8f5e8239d27a27e19386542e2d9bc5bbf2e32f85c892377623ec5d"} Oct 30 18:18:03.955958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:03.955767 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" podStartSLOduration=2.955746394 podStartE2EDuration="2.955746394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:03.885808807 +0000 UTC m=+978.274336925" watchObservedRunningTime="2026-10-30 18:18:03.955746394 +0000 UTC m=+978.344274516" Oct 30 18:18:04.063862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:04.063822 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:04.071493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:04.071423 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:05.068036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.013300 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:05.117072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.108885 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:18:05.551161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.551114 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.552152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.552117 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 18:18:05.584004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.576507 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.837018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.833023 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.854959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.839025 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.973958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.938957 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:05.973958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:05.939060 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.040960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.023014 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.042074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.042032 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:18:06.114166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.114068 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:18:06.136073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.136030 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.156140 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.156093 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.587635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.587589 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.588438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.588416 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:18:06.592266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.592000 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.592860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.592528 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:18:06.856102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.855983 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.865575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:06.865541 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:18:07.190027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.105790 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:18:07.367061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.367026 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:18:07.394191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.132220 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.133267 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.415024 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.415139 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.162055 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.162611 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.355986 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.416809 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.131633 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.162646 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.301726 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.457075 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:18:07.504522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.457176 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:18:07.799013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.795263 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:07.971545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.926614 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.971545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:07.932087 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.359960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:08.334352 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.432961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:08.400234 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Oct 30 18:18:08.450960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:08.450486 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.858194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:08.858147 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.859684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:08.859647 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.888969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:08.869662 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:08.888969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:08.886100 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:18:08.929974 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:18:08.910745 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.111s" Oct 30 18:18:09.675960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.658219 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:18:09.703071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.155329 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:18:09.703071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.155409 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:18:09.759162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.543640 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 18:18:09.820174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.608351 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:18:09.820174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.608410 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:18:09.820174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.609374 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:18:09.905565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.905518 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 18:18:09.913735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.913676 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.084896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:09.999037 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.085578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.075139 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" podUID="c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.100:50052\" within 1s: context deadline exceeded" Oct 30 18:18:10.113295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.113252 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.202702 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:18:10.202648 5624 logging.go:55] [core] [Channel #705 SubChannel #706]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.100:50052", ServerName: "10.134.0.100:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.100:50052: operation was canceled" Oct 30 18:18:10.406976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.280054 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:10.406976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.403023 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 18:18:10.493467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.493406 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.536365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.533054 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:10.690037 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.689855 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.692517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.692459 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.791720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.783435 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:18:10.791720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.783485 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:18:10.791720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.783523 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:18:10.791720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.783566 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:18:10.850922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.850896 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:18:10.963584 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.963511 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:18:10.971531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.971508 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:18:10.980953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:10.980913 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:18:11.307742 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:11.305365 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:11.432973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:11.409954 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:11.528975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:11.515624 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:11.643085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:11.639429 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:11.743638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:11.743578 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:12.805374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:12.805344 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:12.909558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:12.909526 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:13.005783 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:13.005739 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:13.718586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:13.718497 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:15.323188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:15.323119 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:16.891389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:16.891301 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:16.985695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:16.985658 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:17.099957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:17.092308 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:18.540626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:18.540588 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:18:18.546922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:18.546852 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:19.926229 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:18:19.926188 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:18:20.135187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:20.135131 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:21.831138 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:21.831104 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.361034 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:23.360791 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.457523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:23.457485 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.576406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:23.576352 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.841176 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:23.841135 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.945158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:23.945128 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:24.321708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:24.321622 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:25.222100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:25.221999 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:25.494462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:25.494363 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:27.254160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:27.254121 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:28.908446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:28.908410 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk"] Oct 30 18:18:28.913525 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:28.913504 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:28.934110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:28.934080 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk"] Oct 30 18:18:28.940294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:28.940270 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f6a224e7-serving-cert\"" Oct 30 18:18:28.982650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:28.982590 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:29.040574 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.040532 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7340928-a2ba-4a8c-923e-26eb14aafb8d-openshift-service-ca\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.040874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.040853 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-metrics-certs\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.041007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.040899 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92tr\" (UniqueName: \"kubernetes.io/projected/f7340928-a2ba-4a8c-923e-26eb14aafb8d-kube-api-access-r92tr\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.041088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.041047 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-maas-api-tls\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.064622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.064587 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:29.144309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.144268 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-maas-api-tls\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.146501 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.146468 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7340928-a2ba-4a8c-923e-26eb14aafb8d-openshift-service-ca\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.147038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.147000 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-metrics-certs\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.147596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.147557 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7340928-a2ba-4a8c-923e-26eb14aafb8d-openshift-service-ca\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.147596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.147574 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92tr\" (UniqueName: \"kubernetes.io/projected/f7340928-a2ba-4a8c-923e-26eb14aafb8d-kube-api-access-r92tr\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.161835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.161731 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-maas-api-tls\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.165111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.165059 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-metrics-certs\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.203903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.203860 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92tr\" (UniqueName: \"kubernetes.io/projected/f7340928-a2ba-4a8c-923e-26eb14aafb8d-kube-api-access-r92tr\") pod \"maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.233860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.232283 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:29.373956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.373471 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:29.684540 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:18:29.684477 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7340928_a2ba_4a8c_923e_26eb14aafb8d.slice/crio-3a19a14786d5252f5e72b1dac1667d567734226b1f1e72eaa89d13b6df562064 WatchSource:0}: Error finding container 3a19a14786d5252f5e72b1dac1667d567734226b1f1e72eaa89d13b6df562064: Status 404 returned error can't find the container with id 3a19a14786d5252f5e72b1dac1667d567734226b1f1e72eaa89d13b6df562064 Oct 30 18:18:29.685817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.685796 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk"] Oct 30 18:18:29.782163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:29.782127 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" event={"ID":"f7340928-a2ba-4a8c-923e-26eb14aafb8d","Type":"ContainerStarted","Data":"3a19a14786d5252f5e72b1dac1667d567734226b1f1e72eaa89d13b6df562064"} Oct 30 18:18:30.795281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:30.795219 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" event={"ID":"f7340928-a2ba-4a8c-923e-26eb14aafb8d","Type":"ContainerStarted","Data":"5023ac7f299fd322975abe4fbd403517dbece036522cc5ec6e028526e25f6feb"} Oct 30 18:18:30.795781 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:30.795330 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:30.841894 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:30.841844 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" podStartSLOduration=2.51467129 podStartE2EDuration="2.841831098s" totalImagesPullingTime="327.159808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:30.839882748 +0000 UTC m=+1005.228410868" watchObservedRunningTime="2026-10-30 18:18:30.841831098 +0000 UTC m=+1005.230359215" Oct 30 18:18:31.078051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:31.074444 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:31.204454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:31.204325 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:31.300689 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:31.300615 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:31.414477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:31.407686 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:31.735014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:31.731694 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:31.893981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:31.893536 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:31.989813 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:31.989291 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:32.294867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:32.294783 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:33.217646 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:33.217611 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.018005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:35.017969 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.030370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:35.030343 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv"] Oct 30 18:18:35.042493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:35.042468 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352e465b-4vfjv"] Oct 30 18:18:35.196014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:35.195972 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.308797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:35.308704 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.412509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:35.412200 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:35.793632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:35.793599 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1509c67-2df2-4bca-bf50-53e16c3c7059" path="/var/lib/kubelet/pods/f1509c67-2df2-4bca-bf50-53e16c3c7059/volumes" Oct 30 18:18:36.787365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:36.787335 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:36.888899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:36.888874 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:18:36.959954 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:36.959785 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:37.042597 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:37.042517 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:38.513984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:38.513644 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:38.713961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:38.710143 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:38.800416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:38.800314 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:39.208433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:39.208394 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:40.985019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:40.984817 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:41.318048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:41.317927 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:41.680955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:41.662721 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr"] Oct 30 18:18:42.042020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:41.998284 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.085109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:42.085044 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.260962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:42.225821 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:42.395502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:42.395454 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.396649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:42.396622 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:42.754022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:42.720272 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50df807-dc42-412b-8264-2ddc87c48176-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.070206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.070112 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.092268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.092227 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.092564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.092541 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50df807-dc42-412b-8264-2ddc87c48176-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.092768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.092749 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.092900 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.092884 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.114009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.113978 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.114286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.114262 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.114539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.114520 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjnnw\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-kube-api-access-gjnnw\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.114856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:42.952256 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b2b596-openshift-default-dockercfg-x4hhc\"" Oct 30 18:18:43.115003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.079759 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:18:43.146970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:42.756327 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.166961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:42.862303 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.166961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.059709 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:18:43.278529 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:18:43.235508 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.076s" Oct 30 18:18:43.278529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.235627 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr"] Oct 30 18:18:43.278529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.239232 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:43.278529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.078896 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:18:43.278529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.275020 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.278529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.112026 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.278529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.276688 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:18:43.278529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.276756 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:18:43.340095 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:18:43.340007 5624 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50df807_dc42_412b_8264_2ddc87c48176.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:18:43.411972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.408710 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjnnw\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-kube-api-access-gjnnw\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.411972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.408799 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50df807-dc42-412b-8264-2ddc87c48176-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.542975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.534424 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.542975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.534592 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.542975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.534658 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50df807-dc42-412b-8264-2ddc87c48176-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.542975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.534744 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.542975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.534774 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.542975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.534826 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.542975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.534867 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.542975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.535383 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.608111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.608019 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:18:43.739958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.613529 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:18:43.744821 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.646364 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.745043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.646527 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:43.745198 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:43.648317 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:44.119091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.119052 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.151046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.128395 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:18:44.191978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.146832 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50df807-dc42-412b-8264-2ddc87c48176-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:44.191978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.147571 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.191978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.175539 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:18:44.193404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.193350 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.194291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.194262 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:18:44.225457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.225429 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:18:44.230201 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.230179 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:18:44.299262 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.252623 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:44.305143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.279876 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:44.305377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.305029 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.305823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.305072 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:18:44.327411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.307536 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:18:44.398073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.398025 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.489743 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.489712 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50df807-dc42-412b-8264-2ddc87c48176-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:44.532961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.508161 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjnnw\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-kube-api-access-gjnnw\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:44.620638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.620591 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.743899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:44.743792 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:45.130797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.130757 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:18:45.185923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.185878 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.231302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.231259 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:18:45.239961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.234486 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.326975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.326899 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.338961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.335215 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:45.422187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.422142 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.516875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.516834 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.663122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.663017 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.965116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:45.936209 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded" Oct 30 18:18:46.254076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:46.250243 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.254076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:46.251159 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.335124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:46.310881 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.552670 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:46.552440 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:46.624061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:46.618419 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:18:46.625429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:46.625355 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:47.233782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.229105 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.233782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.052576 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.61:50052\" within 1s: context deadline exceeded" Oct 30 18:18:47.233782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.122066 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.233782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.173473 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:18:47.240392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.240254 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Oct 30 18:18:47.243805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:46.957183 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:18:47.246121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.212015 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.040415687s: [/var/lib/containers/storage/overlay/b402f4057d8b6277d783da2f834093ce011c067fe2b38dca32991eb13c9b7ba9/diff /var/log/pods/openshift-monitoring_thanos-querier-74f988c6b9-jw4rr_269d1132-4f1d-4b95-b545-2b2659d245a7/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:47.253091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.252893 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.254959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.254827 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.258003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.228147 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:18:47.765151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.741763 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.189607211s: [/var/lib/containers/storage/overlay/0a951e90a8bd2bf9c126e52a63493f22ed1db4d871aa05916e197a7ed2a23140/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-k8mxn_5f811312-1c2a-4099-9cbe-efe1c5063b66/tuned/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:47.785989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.767784 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.170607864s: [/var/lib/containers/storage/overlay/3e9872a53c61296ace31fec90fe5f3bf967f729bfd4c8470736e623331d49f2b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g74ls_82127ba1-4bef-43cd-8550-719a4402fcab/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:47.796113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.786766 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.823040396s: [/var/lib/containers/storage/overlay/134f2bd157401b25adbc7e733bbaf39965063d9ebe09d941e270f1cc855bc298/diff /var/log/pods/openshift-monitoring_thanos-querier-74f988c6b9-jw4rr_269d1132-4f1d-4b95-b545-2b2659d245a7/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:48.200861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:47.563831 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:49.668996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.662720 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:18:49.668996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.267295 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:18:49.681652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.330442 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Oct 30 18:18:49.681652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.493824 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.681652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.534187 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:18:49.692570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.534304 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:18:49.692570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.573904 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-5f78687d5b-j6n87" podUID="d13122e5-4950-40fc-8bb5-046b9fa77766" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.54:50052\" within 1s: context deadline exceeded" Oct 30 18:18:49.692570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.574040 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.692570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.606139 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.716194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.645401 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:18:49.720957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.646083 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:18:49.743601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.646654 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:18:49.745760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.667623 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" podUID="54747d07-9037-4c21-be5c-c554f0b21b5d" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.745760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.793036 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.753193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.751096 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.248920 5624 trace.go:236] Trace[1733239413]: "iptables ChainExists" (30-Oct-2026 18:18:45.898) (total time: 2350ms): Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: Trace[1733239413]: [2.350747403s] [2.350747403s] END Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.277502 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.894560382s: [/var/lib/containers/storage/overlay/67eb067ac534d92b968e5556159e9b637ea6ea6cd07c68ae60f09111feb11c60/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-brcll_e2a6cc3b-391d-4626-908a-5466994e21d5/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.278852 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.321244994s: [/var/lib/containers/storage/overlay/d2ada5068a38c5b5cacf8c651cc51d8124c72f438216acd9cac8348cb55c3f82/diff /var/log/pods/openshift-image-registry_node-ca-l6vjs_c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2/node-ca/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:18:48.527601 5624 logging.go:55] [core] [Channel #779 SubChannel #780]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.61:50052", ServerName: "10.134.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:43984->10.134.0.61:50052: use of closed network connection" Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.700248 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.171318116s: [/var/lib/containers/storage/overlay/cce3fecf6b2e83604a8dfb551fc71e3a777a73065258bd85cea2b73f8bcf6c3e/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g74ls_82127ba1-4bef-43cd-8550-719a4402fcab/nbdb/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.792204 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.757587 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.952863 5624 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-f87jc container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9443/health\": context deadline exceeded" start-of-body= Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.757681 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" podUID="9e62ca62-c183-4859-8d52-a52aa8182c73" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.12:9443/health\": context deadline exceeded" Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.279655 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.833097483s: [/var/lib/containers/storage/overlay/e619ab02e5347c29da3175bb6eabaf354c177e7a2b3ac97c4e1c7258c89a5174/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:18:49.308807 5624 logging.go:55] [core] [Channel #782 SubChannel #783]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.54:50052", ServerName: "10.134.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.54:50052: operation was canceled" Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.648429 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.085676284s: [/var/lib/containers/storage/overlay/b1b9c1884eb73efff58ab50a1e3d9a6588147e0ade902919ef9794d5ae631b82/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pjv29_bf49c75c-fb11-4c2d-ac07-b41d85f8f106/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.661090 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.757757 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.768018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.758452 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.877505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.772466 5624 patch_prober.go:29] interesting pod/thanos-querier-74f988c6b9-jw4rr container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:49.877505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.772539 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" podUID="269d1132-4f1d-4b95-b545-2b2659d245a7" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.11:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.877505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.815533 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:49.877505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.815617 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.877505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.873101 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.968491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:49.968285 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" podUID="09728775-6be0-45a5-bfb9-480ff7a90759" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.175063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.171804 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:50.175063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.172015 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.236271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:48.952721 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Oct 30 18:18:50.255487 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.239398 5624 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:50.284740 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.284593 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.378560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.256140 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" podUID="f7340928-a2ba-4a8c-923e-26eb14aafb8d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.103:8443/health\": context deadline exceeded" Oct 30 18:18:50.379295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.344863 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 18:18:50.390309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.390174 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.766962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.757871 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.774951 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.774893 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.000583529s: [/var/lib/containers/storage/overlay/253c9b4719dcd12f7733db13e76ceaa594633cbd6c25ecd8a56191e0fd07e97f/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:50.791594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.775231 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.798540 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.798500 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.024182856s: [/var/lib/containers/storage/overlay/5aef074ad5c30cac3f0b916a49cf7842b9e9ce6776cca136256e9dd57482ac47/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:51.362390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.918207 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.143886415s: [/var/lib/containers/storage/overlay/ef571626bb8ec813f1ef7ff165682c857eb60aa8ef7c05b45e382c46503698d0/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 18:18:51.362390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.945514 5624 trace.go:236] Trace[788481640]: "Calculate volume metrics of istio-token for pod openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" (30-Oct-2026 18:18:49.760) (total time: 1184ms): Oct 30 18:18:51.362390 ip-10-0-133-47 kubenswrapper[5624]: Trace[788481640]: [1.184701728s] [1.184701728s] END Oct 30 18:18:51.362390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.251023 5624 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:51.362390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.335074 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:51.362390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.291209 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:18:51.362390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.863155 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:51.384966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.863188 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:51.384966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.938026 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" podUID="5c4422bc-c3e1-48f1-99a8-c62260e1d25c" containerName="limitador" probeResult="failure" output="Get \"http://10.134.0.36:8080/status\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:51.384966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.128092 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.62:50052\" within 1s: context deadline exceeded" Oct 30 18:18:51.384966 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:18:51.291173 5624 logging.go:55] [core] [Channel #785 SubChannel #788]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.62:50052", ServerName: "10.134.0.62:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.62:50052: operation was canceled" Oct 30 18:18:51.416908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.416853 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:51.448456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.448408 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 30 18:18:51.462462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.462415 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:51.467077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.467015 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:51.532450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.509477 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:18:51.818041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:50.937792 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:51.818041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.643298 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:18:51.818041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.643338 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:18:51.818041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.643385 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:18:51.818041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.643411 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:18:51.818041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.662503 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:18:51.863170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:51.863117 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" podUID="c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.100:50052\" within 1s: context deadline exceeded" Oct 30 18:18:51.980968 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:18:51.971535 5624 logging.go:55] [core] [Channel #784 SubChannel #790]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.100:50052", ServerName: "10.134.0.100:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.100:50052: operation was canceled" Oct 30 18:18:52.033313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.033254 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:52.359887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.057401 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:18:52.360621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.360561 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:18:52.361008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.360974 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:18:52.361672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.350678 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:52.362159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.359843 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:52.421981 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:18:52.402584 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="6.594s" Oct 30 18:18:52.470254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.470181 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:52.474368 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:18:52.474302 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.646559 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.646683 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" event={"ID":"d50df807-dc42-412b-8264-2ddc87c48176","Type":"ContainerStarted","Data":"ec62f99bc031888e3cb6453f4cd87485484d87fe8e7b036b2a63ada4d46dcec0"} Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.646796 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.646836 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr"] Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.646905 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.646987 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647039 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647093 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647135 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647263 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647463 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647525 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647578 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647626 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647665 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647707 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647757 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:18:52.650028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:52.647802 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:18:53.558765 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:53.558544 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" event={"ID":"d50df807-dc42-412b-8264-2ddc87c48176","Type":"ContainerStarted","Data":"e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325"} Oct 30 18:18:53.741883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:53.741681 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podStartSLOduration=12.741637761 podStartE2EDuration="12.741637761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:53.683666672 +0000 UTC m=+1028.072194815" watchObservedRunningTime="2026-10-30 18:18:53.741637761 +0000 UTC m=+1028.130165879" Oct 30 18:18:53.750083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:53.750021 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:53.756339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:53.756259 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:18:53.756757 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:53.756381 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 18:18:54.781047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:54.772114 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:54.784917 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:18:54.784828 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.002s" Oct 30 18:18:54.981153 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:54.981074 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:55.998430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:55.998346 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:56.002773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:56.002708 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:18:56.136241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:56.136035 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:56.263561 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:56.261269 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:56.462772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:56.462671 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.143975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.143861 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.241656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.241601 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.331965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.329368 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5"] Oct 30 18:18:58.336564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.336535 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:18:58.361887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.358741 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5"] Oct 30 18:18:58.423246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.369104 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b2b596-dockercfg-6vvdt\"" Oct 30 18:18:58.423246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.369206 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b2b596\"" Oct 30 18:18:58.522959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.478264 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0ef545-bf5b-4665-95a8-2c6b578138a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5\" (UID: \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:18:58.539962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.523493 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qmms\" (UniqueName: \"kubernetes.io/projected/8c0ef545-bf5b-4665-95a8-2c6b578138a3-kube-api-access-8qmms\") pod \"payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5\" (UID: \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:18:58.658007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.655812 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.736861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.733422 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0ef545-bf5b-4665-95a8-2c6b578138a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5\" (UID: \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:18:58.736861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.733699 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qmms\" (UniqueName: \"kubernetes.io/projected/8c0ef545-bf5b-4665-95a8-2c6b578138a3-kube-api-access-8qmms\") pod \"payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5\" (UID: \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:18:58.744979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.743483 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0ef545-bf5b-4665-95a8-2c6b578138a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5\" (UID: \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:18:58.783445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.783404 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4"] Oct 30 18:18:58.803320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.803256 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4"] Oct 30 18:18:58.803896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.803873 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:18:58.839058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.833404 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qmms\" (UniqueName: \"kubernetes.io/projected/8c0ef545-bf5b-4665-95a8-2c6b578138a3-kube-api-access-8qmms\") pod \"payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5\" (UID: \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:18:58.958295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.955522 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhrl\" (UniqueName: \"kubernetes.io/projected/2d81034f-ac92-46ff-a8d6-a432835b1426-kube-api-access-pfhrl\") pod \"payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4\" (UID: \"2d81034f-ac92-46ff-a8d6-a432835b1426\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:18:58.958295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:58.955905 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d81034f-ac92-46ff-a8d6-a432835b1426-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4\" (UID: \"2d81034f-ac92-46ff-a8d6-a432835b1426\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:18:59.002042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:59.001140 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:18:59.057600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:59.057534 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d81034f-ac92-46ff-a8d6-a432835b1426-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4\" (UID: \"2d81034f-ac92-46ff-a8d6-a432835b1426\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:18:59.057854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:59.057751 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhrl\" (UniqueName: \"kubernetes.io/projected/2d81034f-ac92-46ff-a8d6-a432835b1426-kube-api-access-pfhrl\") pod \"payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4\" (UID: \"2d81034f-ac92-46ff-a8d6-a432835b1426\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:18:59.060114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:59.059641 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d81034f-ac92-46ff-a8d6-a432835b1426-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4\" (UID: \"2d81034f-ac92-46ff-a8d6-a432835b1426\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:18:59.095985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:59.095664 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhrl\" (UniqueName: \"kubernetes.io/projected/2d81034f-ac92-46ff-a8d6-a432835b1426-kube-api-access-pfhrl\") pod \"payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4\" (UID: \"2d81034f-ac92-46ff-a8d6-a432835b1426\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:18:59.202632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:59.202513 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:18:59.382260 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:18:59.382190 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0ef545_bf5b_4665_95a8_2c6b578138a3.slice/crio-560e5cad8e9ac723fc6c2514c8137e18400170c6b9e06efc41c3a86ed1af29cc WatchSource:0}: Error finding container 560e5cad8e9ac723fc6c2514c8137e18400170c6b9e06efc41c3a86ed1af29cc: Status 404 returned error can't find the container with id 560e5cad8e9ac723fc6c2514c8137e18400170c6b9e06efc41c3a86ed1af29cc Oct 30 18:18:59.383577 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:59.383468 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5"] Oct 30 18:18:59.587889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:18:59.587800 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4"] Oct 30 18:19:00.049794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.049752 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" event={"ID":"2d81034f-ac92-46ff-a8d6-a432835b1426","Type":"ContainerStarted","Data":"15892bb4ec4a425073be3198b8273086186c55d7246007db983e66248e0c6853"} Oct 30 18:19:00.050029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.049828 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" event={"ID":"2d81034f-ac92-46ff-a8d6-a432835b1426","Type":"ContainerStarted","Data":"a2a05a15e0bc72daf2f8459dc13b2bea004ef455bcf0a3d21385ef2c4c93210d"} Oct 30 18:19:00.050029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.049864 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:19:00.051876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.051845 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" event={"ID":"8c0ef545-bf5b-4665-95a8-2c6b578138a3","Type":"ContainerStarted","Data":"1bbc97b10fc4aafadb02756054dcd8d507850108d79608ea1a48a36e48282935"} Oct 30 18:19:00.052030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.051902 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" event={"ID":"8c0ef545-bf5b-4665-95a8-2c6b578138a3","Type":"ContainerStarted","Data":"560e5cad8e9ac723fc6c2514c8137e18400170c6b9e06efc41c3a86ed1af29cc"} Oct 30 18:19:00.052030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.051976 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:19:00.092587 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.092520 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" podStartSLOduration=2.092498244 podStartE2EDuration="2.092498244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:00.089240159 +0000 UTC m=+1034.477768274" watchObservedRunningTime="2026-10-30 18:19:00.092498244 +0000 UTC m=+1034.481026363" Oct 30 18:19:00.132017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.131912 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" podStartSLOduration=2.13189015 podStartE2EDuration="2.13189015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:00.129774633 +0000 UTC m=+1034.518302753" watchObservedRunningTime="2026-10-30 18:19:00.13189015 +0000 UTC m=+1034.520418268" Oct 30 18:19:00.361169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.359761 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:00.591122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.591078 5624 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:19:00.591310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:00.591170 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:01.681851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:01.681822 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr"] Oct 30 18:19:01.726916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:01.726876 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w"] Oct 30 18:19:01.757439 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:01.757380 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:01.863216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:01.863174 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:01.995412 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:01.995213 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:03.146925 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:03.146894 5624 generic.go:356] "Generic (PLEG): container finished" podID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerID="22ff70dbf386fbec9d2297213111f751f1b7facb13f22cd3cc460fb84b38dfac" exitCode=0 Oct 30 18:19:03.147368 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:03.146979 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" event={"ID":"23496535-1b6c-4a4d-8080-a06ce5a26b93","Type":"ContainerDied","Data":"22ff70dbf386fbec9d2297213111f751f1b7facb13f22cd3cc460fb84b38dfac"} Oct 30 18:19:03.147459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:03.147441 5624 scope.go:122] "RemoveContainer" containerID="22ff70dbf386fbec9d2297213111f751f1b7facb13f22cd3cc460fb84b38dfac" Oct 30 18:19:03.149008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:03.148981 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:03.149131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:03.149046 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:03.149131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:03.149079 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:04.158197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:04.158166 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" event={"ID":"23496535-1b6c-4a4d-8080-a06ce5a26b93","Type":"ContainerStarted","Data":"7445262021df262b8623ffe205147d2b1546cc18aac2cc601a9b5924fabdf8b9"} Oct 30 18:19:04.348201 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:04.348164 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:19:04.348367 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:04.348338 5624 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:19:04.348410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:04.348378 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:19:04.652047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:04.652008 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:05.349033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:05.348994 5624 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:19:05.349492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:05.349062 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:19:06.348317 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.348283 5624 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:19:06.348515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.348340 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:19:06.526424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.526384 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57"] Oct 30 18:19:06.531093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.531076 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.553009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.552982 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57"] Oct 30 18:19:06.562980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.559145 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a90958fa-serving-cert\"" Oct 30 18:19:06.617237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.617149 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:06.637475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.637443 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj955\" (UniqueName: \"kubernetes.io/projected/cec76070-478d-41dd-b3f6-9cb678ac839c-kube-api-access-zj955\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.637752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.637727 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec76070-478d-41dd-b3f6-9cb678ac839c-openshift-service-ca\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.637886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.637870 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.637948 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.637902 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-metrics-certs\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.682741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.682706 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" podUID="c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" containerName="payload-processing" containerID="cri-o://d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328" gracePeriod=25 Oct 30 18:19:06.727042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.724910 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:06.741302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.740697 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" podUID="6d612bfa-5e32-4d59-8f30-40bef0c04c20" containerName="payload-pre-processing" containerID="cri-o://abeb34719bfc4ba4ef2e41545b95f557cbb33adc7586f05ece43bea1b45be307" gracePeriod=25 Oct 30 18:19:06.746875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.743116 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.746875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.743194 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-metrics-certs\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.746875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.743317 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj955\" (UniqueName: \"kubernetes.io/projected/cec76070-478d-41dd-b3f6-9cb678ac839c-kube-api-access-zj955\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.746875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.743587 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec76070-478d-41dd-b3f6-9cb678ac839c-openshift-service-ca\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.746875 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:19:06.743319 5624 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-serving-cert: secret "maas-api-e2e-derive-a90958fa-serving-cert" not found Oct 30 18:19:06.746875 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:19:06.744169 5624 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls podName:cec76070-478d-41dd-b3f6-9cb678ac839c nodeName:}" failed. No retries permitted until 2026-10-30 18:19:07.244146775 +0000 UTC m=+1041.632674888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls") pod "maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" (UID: "cec76070-478d-41dd-b3f6-9cb678ac839c") : secret "maas-api-e2e-derive-a90958fa-serving-cert" not found Oct 30 18:19:06.746875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.744455 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec76070-478d-41dd-b3f6-9cb678ac839c-openshift-service-ca\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.758703 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.758670 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-metrics-certs\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.768704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.768676 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj955\" (UniqueName: \"kubernetes.io/projected/cec76070-478d-41dd-b3f6-9cb678ac839c-kube-api-access-zj955\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:06.908867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:06.908790 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:07.227472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.227440 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:19:07.228895 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.228867 5624 generic.go:356] "Generic (PLEG): container finished" podID="6d612bfa-5e32-4d59-8f30-40bef0c04c20" containerID="abeb34719bfc4ba4ef2e41545b95f557cbb33adc7586f05ece43bea1b45be307" exitCode=0 Oct 30 18:19:07.229076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.229008 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" event={"ID":"6d612bfa-5e32-4d59-8f30-40bef0c04c20","Type":"ContainerDied","Data":"abeb34719bfc4ba4ef2e41545b95f557cbb33adc7586f05ece43bea1b45be307"} Oct 30 18:19:07.231147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.231117 5624 generic.go:356] "Generic (PLEG): container finished" podID="c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" containerID="d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328" exitCode=0 Oct 30 18:19:07.231392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.231370 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" event={"ID":"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc","Type":"ContainerDied","Data":"d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328"} Oct 30 18:19:07.231485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.231425 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" event={"ID":"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc","Type":"ContainerDied","Data":"3d041e9051b03009657987044e903bcd1a3517ab78bce1474fb8b930ad30f96c"} Oct 30 18:19:07.231485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.231459 5624 scope.go:122] "RemoveContainer" containerID="d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328" Oct 30 18:19:07.231623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.231608 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr" Oct 30 18:19:07.244399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.244377 5624 scope.go:122] "RemoveContainer" containerID="d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328" Oct 30 18:19:07.244708 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:19:07.244679 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328\": container with ID starting with d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328 not found: ID does not exist" containerID="d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328" Oct 30 18:19:07.244829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.244711 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328"} err="failed to get container status \"d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328\": rpc error: code = NotFound desc = could not find container \"d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328\": container with ID starting with d67c916677afe9f4607e49fff232746c0cf23f6ed9bdcd3b7de277408adad328 not found: ID does not exist" Oct 30 18:19:07.252989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.252966 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:07.255377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.255357 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls\") pod \"maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:07.256020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.256002 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:19:07.350847 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.350816 5624 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:19:07.351121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.351086 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:19:07.354081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.354054 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-plugins-config-volume\") pod \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\" (UID: \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\") " Oct 30 18:19:07.354248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.354203 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d612bfa-5e32-4d59-8f30-40bef0c04c20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d612bfa-5e32-4d59-8f30-40bef0c04c20-plugins-config-volume\") pod \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\" (UID: \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\") " Oct 30 18:19:07.354248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.354235 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-kube-api-access-h6sv2\" (UniqueName: \"kubernetes.io/projected/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-kube-api-access-h6sv2\") pod \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\" (UID: \"c30c309c-be3b-4a8a-83ed-8fcae7cb13bc\") " Oct 30 18:19:07.354538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.354514 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-plugins-config-volume" pod "c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" (UID: "c30c309c-be3b-4a8a-83ed-8fcae7cb13bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:07.354832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.354813 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d612bfa-5e32-4d59-8f30-40bef0c04c20-plugins-config-volume" pod "6d612bfa-5e32-4d59-8f30-40bef0c04c20" (UID: "6d612bfa-5e32-4d59-8f30-40bef0c04c20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:07.355113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.355094 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d612bfa-5e32-4d59-8f30-40bef0c04c20-kube-api-access-22277\" (UniqueName: \"kubernetes.io/projected/6d612bfa-5e32-4d59-8f30-40bef0c04c20-kube-api-access-22277\") pod \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\" (UID: \"6d612bfa-5e32-4d59-8f30-40bef0c04c20\") " Oct 30 18:19:07.355981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.355857 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d612bfa-5e32-4d59-8f30-40bef0c04c20-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:07.355981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.355890 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:07.357584 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.357563 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-kube-api-access-h6sv2" pod "c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" (UID: "c30c309c-be3b-4a8a-83ed-8fcae7cb13bc"). InnerVolumeSpecName "kube-api-access-h6sv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:07.359844 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.359824 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d612bfa-5e32-4d59-8f30-40bef0c04c20-kube-api-access-22277" pod "6d612bfa-5e32-4d59-8f30-40bef0c04c20" (UID: "6d612bfa-5e32-4d59-8f30-40bef0c04c20"). InnerVolumeSpecName "kube-api-access-22277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:07.445576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.445494 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:07.464624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.458442 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22277\" (UniqueName: \"kubernetes.io/projected/6d612bfa-5e32-4d59-8f30-40bef0c04c20-kube-api-access-22277\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:07.464624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.458467 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6sv2\" (UniqueName: \"kubernetes.io/projected/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc-kube-api-access-h6sv2\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:07.676089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.676019 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr"] Oct 30 18:19:07.683441 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.682775 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-535b743b-c687fddc4-zzrvr"] Oct 30 18:19:07.830965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:07.830617 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" path="/var/lib/kubelet/pods/c30c309c-be3b-4a8a-83ed-8fcae7cb13bc/volumes" Oct 30 18:19:08.069274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.068964 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:08.070073 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:19:08.070024 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec76070_478d_41dd_b3f6_9cb678ac839c.slice/crio-1e8a5f51f4bc532e01943397c8ed70dce5b1880cba33a3fd61a92885256c56db WatchSource:0}: Error finding container 1e8a5f51f4bc532e01943397c8ed70dce5b1880cba33a3fd61a92885256c56db: Status 404 returned error can't find the container with id 1e8a5f51f4bc532e01943397c8ed70dce5b1880cba33a3fd61a92885256c56db Oct 30 18:19:08.071302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.071250 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57"] Oct 30 18:19:08.160200 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.160174 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:08.265023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.264975 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" Oct 30 18:19:08.265023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.264929 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w" event={"ID":"6d612bfa-5e32-4d59-8f30-40bef0c04c20","Type":"ContainerDied","Data":"aa157d2f1fd9e3ea7c7c2560ac182135e1cf1785e9103e80efed789f4e3bb000"} Oct 30 18:19:08.265289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.265082 5624 scope.go:122] "RemoveContainer" containerID="abeb34719bfc4ba4ef2e41545b95f557cbb33adc7586f05ece43bea1b45be307" Oct 30 18:19:08.269454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.269202 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" event={"ID":"cec76070-478d-41dd-b3f6-9cb678ac839c","Type":"ContainerStarted","Data":"1e8a5f51f4bc532e01943397c8ed70dce5b1880cba33a3fd61a92885256c56db"} Oct 30 18:19:08.353081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.350347 5624 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:19:08.353081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.350407 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:19:08.367960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.363609 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w"] Oct 30 18:19:08.429988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.426112 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-535b743b-657bf56d4c-6nx7w"] Oct 30 18:19:08.949290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:08.949158 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:09.282090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:09.281686 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" event={"ID":"cec76070-478d-41dd-b3f6-9cb678ac839c","Type":"ContainerStarted","Data":"415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07"} Oct 30 18:19:09.282090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:09.282009 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:09.343162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:09.343084 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" podStartSLOduration=2.8658032589999998 podStartE2EDuration="3.343069345s" totalImagesPullingTime="477.266086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:09.330238672 +0000 UTC m=+1043.718766790" watchObservedRunningTime="2026-10-30 18:19:09.343069345 +0000 UTC m=+1043.731597461" Oct 30 18:19:09.348524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:09.348500 5624 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 18:19:09.348727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:09.348706 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 18:19:09.793781 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:09.793750 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d612bfa-5e32-4d59-8f30-40bef0c04c20" path="/var/lib/kubelet/pods/6d612bfa-5e32-4d59-8f30-40bef0c04c20/volumes" Oct 30 18:19:10.352004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:10.351975 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:19:10.352387 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:10.352348 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:19:10.353044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:10.353021 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:19:10.504232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:10.503187 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.027441 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:11.027409 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.065550 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:11.065526 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:19:11.065694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:11.065595 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:19:11.144674 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:11.144640 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.233525 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:11.233497 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:11.351646 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:11.351568 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.002311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.002282 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.418275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.418242 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr"] Oct 30 18:19:12.419423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.419396 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6d612bfa-5e32-4d59-8f30-40bef0c04c20" containerName="payload-pre-processing" Oct 30 18:19:12.419423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.419421 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d612bfa-5e32-4d59-8f30-40bef0c04c20" containerName="payload-pre-processing" Oct 30 18:19:12.419593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.419470 5624 cpu_manager.go:414] "containerMap: removing container" podUID="c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" containerName="payload-processing" Oct 30 18:19:12.419593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.419478 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" containerName="payload-processing" Oct 30 18:19:12.419705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.419650 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30c309c-be3b-4a8a-83ed-8fcae7cb13bc" containerName="payload-processing" Oct 30 18:19:12.419705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.419671 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d612bfa-5e32-4d59-8f30-40bef0c04c20" containerName="payload-pre-processing" Oct 30 18:19:12.423856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.423835 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.442850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.442793 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr"] Oct 30 18:19:12.449448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.449161 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b0171017-openshift-default-dockercfg-m2rqb\"" Oct 30 18:19:12.470422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.470388 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:12.480812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.480782 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-data\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.480980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.480955 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-certs\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.481046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.481014 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-token\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.481113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.481093 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwr76\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-kube-api-access-fwr76\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.481170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.481140 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-credential-socket\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.481415 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.481392 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-socket\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.481484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.481434 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istiod-ca-cert\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.481578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.481564 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-envoy\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.481680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.481666 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-podinfo\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.635959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.629245 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-data\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.635959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.630044 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-certs\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.635959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.630771 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-certs\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.635959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.630826 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-token\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.635959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.630877 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwr76\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-kube-api-access-fwr76\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.635959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.631045 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-data\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.635959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.631094 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-credential-socket\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.639954 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.639729 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-credential-socket\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.658956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.652683 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-socket\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.658956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.652755 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-socket\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.658956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.652799 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istiod-ca-cert\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.658956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.653089 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-envoy\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.658956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.653215 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-podinfo\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.658956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.655845 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istiod-ca-cert\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.671000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.665846 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-podinfo\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.680951 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.676682 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-envoy\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.706956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.699956 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwr76\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-kube-api-access-fwr76\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.738958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.733197 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-token\") pod \"e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.747795 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.746555 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:12.883861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:12.883822 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.162587 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:19:13.162543 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8eafb2b_86dc_4a79_8ff6_948cfcf01b5b.slice/crio-aa0873374b75e5992ad2427581267924a79df5bc2ad996f9cfa0de2718f30ff7 WatchSource:0}: Error finding container aa0873374b75e5992ad2427581267924a79df5bc2ad996f9cfa0de2718f30ff7: Status 404 returned error can't find the container with id aa0873374b75e5992ad2427581267924a79df5bc2ad996f9cfa0de2718f30ff7 Oct 30 18:19:13.170957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.168041 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:13.170957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.168302 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:13.170957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.168441 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:13.170957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.169047 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr"] Oct 30 18:19:13.354092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.352996 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" event={"ID":"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b","Type":"ContainerStarted","Data":"b4096edb54b80e0fbb40604e9788d453d7117a454bdddd89f6438c6449675f97"} Oct 30 18:19:13.354092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.353070 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" event={"ID":"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b","Type":"ContainerStarted","Data":"aa0873374b75e5992ad2427581267924a79df5bc2ad996f9cfa0de2718f30ff7"} Oct 30 18:19:13.399815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.399762 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" podStartSLOduration=1.399748056 podStartE2EDuration="1.399748056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:13.394533886 +0000 UTC m=+1047.783062005" watchObservedRunningTime="2026-10-30 18:19:13.399748056 +0000 UTC m=+1047.788276173" Oct 30 18:19:13.463147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.463063 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.747276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.747200 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:13.915910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:13.915875 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:14.001284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:14.001205 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:14.758210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:14.758180 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:15.002132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:15.002079 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:15.123093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:15.123065 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:15.296089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:15.296060 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:15.376231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:15.376151 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:15.381097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:15.378278 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:19:15.610636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:15.610610 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:16.700552 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:16.700484 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:16.802722 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:16.802686 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:17.427197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:17.427155 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:18.502903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:18.502872 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:18.640543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:18.640481 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:18.718503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:18.718463 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:18.858371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:18.858278 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:18.872338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:18.871734 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p"] Oct 30 18:19:18.872970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:18.872632 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" containerID="cri-o://7445262021df262b8623ffe205147d2b1546cc18aac2cc601a9b5924fabdf8b9" gracePeriod=30 Oct 30 18:19:19.822149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:19.822043 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:19.919741 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:19:19.919706 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:19:19.920078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:19.920053 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:20.021154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:20.021117 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:20.685473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:20.685444 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:20.768076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:20.768052 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:21.056572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:21.056492 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:22.622635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:22.622594 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:23.940798 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:23.940768 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk"] Oct 30 18:19:23.941293 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:23.941005 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" podUID="f7340928-a2ba-4a8c-923e-26eb14aafb8d" containerName="maas-api" containerID="cri-o://5023ac7f299fd322975abe4fbd403517dbece036522cc5ec6e028526e25f6feb" gracePeriod=30 Oct 30 18:19:23.986171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:23.986133 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:24.629235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.629207 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:24.629967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.629928 5624 generic.go:356] "Generic (PLEG): container finished" podID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerID="7445262021df262b8623ffe205147d2b1546cc18aac2cc601a9b5924fabdf8b9" exitCode=0 Oct 30 18:19:24.630258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.630098 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" event={"ID":"23496535-1b6c-4a4d-8080-a06ce5a26b93","Type":"ContainerDied","Data":"7445262021df262b8623ffe205147d2b1546cc18aac2cc601a9b5924fabdf8b9"} Oct 30 18:19:24.630354 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.630308 5624 scope.go:122] "RemoveContainer" containerID="22ff70dbf386fbec9d2297213111f751f1b7facb13f22cd3cc460fb84b38dfac" Oct 30 18:19:24.631821 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.631795 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:19:24.634111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.634084 5624 generic.go:356] "Generic (PLEG): container finished" podID="f7340928-a2ba-4a8c-923e-26eb14aafb8d" containerID="5023ac7f299fd322975abe4fbd403517dbece036522cc5ec6e028526e25f6feb" exitCode=0 Oct 30 18:19:24.634276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.634255 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" event={"ID":"f7340928-a2ba-4a8c-923e-26eb14aafb8d","Type":"ContainerDied","Data":"5023ac7f299fd322975abe4fbd403517dbece036522cc5ec6e028526e25f6feb"} Oct 30 18:19:24.747599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.747570 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-credential-socket\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.747730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.747649 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-socket\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.747730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.747683 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-envoy\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.747730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.747718 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-certs\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.747926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.747754 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-podinfo\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.747926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.747809 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23496535-1b6c-4a4d-8080-a06ce5a26b93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23496535-1b6c-4a4d-8080-a06ce5a26b93-istiod-ca-cert\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.748770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.747963 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-token\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-token\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.748770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.748129 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-kube-api-access-l5wmp\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-kube-api-access-l5wmp\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.748770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.748193 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-data\") pod \"23496535-1b6c-4a4d-8080-a06ce5a26b93\" (UID: \"23496535-1b6c-4a4d-8080-a06ce5a26b93\") " Oct 30 18:19:24.750650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.750139 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-credential-socket" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.750650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.750371 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-socket" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.751606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.751311 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-certs" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.754459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.754302 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23496535-1b6c-4a4d-8080-a06ce5a26b93-istiod-ca-cert" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:24.756347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.755232 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-data" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.759273 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.759218 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-token" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:24.763020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.758765 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-envoy" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:24.785872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.777795 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-kube-api-access-l5wmp" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "kube-api-access-l5wmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:24.785872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.777889 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-podinfo" pod "23496535-1b6c-4a4d-8080-a06ce5a26b93" (UID: "23496535-1b6c-4a4d-8080-a06ce5a26b93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:24.832958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.816919 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:24.856601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856562 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.856601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856601 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5wmp\" (UniqueName: \"kubernetes.io/projected/23496535-1b6c-4a4d-8080-a06ce5a26b93-kube-api-access-l5wmp\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.856829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856623 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.856829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856637 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.856829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856651 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.856829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856664 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.856829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856680 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23496535-1b6c-4a4d-8080-a06ce5a26b93-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.856829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856691 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23496535-1b6c-4a4d-8080-a06ce5a26b93-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.856829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.856704 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23496535-1b6c-4a4d-8080-a06ce5a26b93-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.860612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.860582 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:19:24.958226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.958199 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-maas-api-tls\") pod \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " Oct 30 18:19:24.958634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.958337 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7340928-a2ba-4a8c-923e-26eb14aafb8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7340928-a2ba-4a8c-923e-26eb14aafb8d-openshift-service-ca\") pod \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " Oct 30 18:19:24.958634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.958401 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-metrics-certs\") pod \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " Oct 30 18:19:24.958634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.958466 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7340928-a2ba-4a8c-923e-26eb14aafb8d-kube-api-access-r92tr\" (UniqueName: \"kubernetes.io/projected/f7340928-a2ba-4a8c-923e-26eb14aafb8d-kube-api-access-r92tr\") pod \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\" (UID: \"f7340928-a2ba-4a8c-923e-26eb14aafb8d\") " Oct 30 18:19:24.958811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.958637 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7340928-a2ba-4a8c-923e-26eb14aafb8d-openshift-service-ca" pod "f7340928-a2ba-4a8c-923e-26eb14aafb8d" (UID: "f7340928-a2ba-4a8c-923e-26eb14aafb8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:24.959282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.959255 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7340928-a2ba-4a8c-923e-26eb14aafb8d-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:24.961782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.961752 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-metrics-certs" pod "f7340928-a2ba-4a8c-923e-26eb14aafb8d" (UID: "f7340928-a2ba-4a8c-923e-26eb14aafb8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:24.961952 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.961899 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-maas-api-tls" pod "f7340928-a2ba-4a8c-923e-26eb14aafb8d" (UID: "f7340928-a2ba-4a8c-923e-26eb14aafb8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:24.964171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:24.964149 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7340928-a2ba-4a8c-923e-26eb14aafb8d-kube-api-access-r92tr" pod "f7340928-a2ba-4a8c-923e-26eb14aafb8d" (UID: "f7340928-a2ba-4a8c-923e-26eb14aafb8d"). InnerVolumeSpecName "kube-api-access-r92tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:25.060335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.060303 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.060335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.060334 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r92tr\" (UniqueName: \"kubernetes.io/projected/f7340928-a2ba-4a8c-923e-26eb14aafb8d-kube-api-access-r92tr\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.060542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.060345 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7340928-a2ba-4a8c-923e-26eb14aafb8d-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.197149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.197119 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:25.643842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.643812 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" Oct 30 18:19:25.644059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.643846 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk" event={"ID":"f7340928-a2ba-4a8c-923e-26eb14aafb8d","Type":"ContainerDied","Data":"3a19a14786d5252f5e72b1dac1667d567734226b1f1e72eaa89d13b6df562064"} Oct 30 18:19:25.644059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.643910 5624 scope.go:122] "RemoveContainer" containerID="5023ac7f299fd322975abe4fbd403517dbece036522cc5ec6e028526e25f6feb" Oct 30 18:19:25.645739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.645721 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" Oct 30 18:19:25.645827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.645718 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p" event={"ID":"23496535-1b6c-4a4d-8080-a06ce5a26b93","Type":"ContainerDied","Data":"f657e72bf50130c6a949b6847f04fc1629cd0245a41c5a5650a076bdf0b0d1b8"} Oct 30 18:19:25.657443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.657421 5624 scope.go:122] "RemoveContainer" containerID="7445262021df262b8623ffe205147d2b1546cc18aac2cc601a9b5924fabdf8b9" Oct 30 18:19:25.730842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.730809 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p"] Oct 30 18:19:25.735011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.734988 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-zmh2p"] Oct 30 18:19:25.767632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.767603 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk"] Oct 30 18:19:25.772021 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.771998 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f6a224e7-5c8585cd96-fp9lk"] Oct 30 18:19:25.795658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.795632 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" path="/var/lib/kubelet/pods/23496535-1b6c-4a4d-8080-a06ce5a26b93/volumes" Oct 30 18:19:25.796231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:25.796215 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7340928-a2ba-4a8c-923e-26eb14aafb8d" path="/var/lib/kubelet/pods/f7340928-a2ba-4a8c-923e-26eb14aafb8d/volumes" Oct 30 18:19:26.462897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:26.462805 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:26.479986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:26.479950 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf"] Oct 30 18:19:26.480206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:26.480179 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" podUID="647187d5-0217-4a90-a5c7-900553dfd802" containerName="istio-proxy" containerID="cri-o://f1276e6dbc8f5e8239d27a27e19386542e2d9bc5bbf2e32f85c892377623ec5d" gracePeriod=30 Oct 30 18:19:26.498590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:26.498556 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:19:27.325907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:27.325875 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:27.440155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:27.438781 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:27.543879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:27.543845 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:29.992106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:29.992073 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:32.088092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.088055 5624 generic.go:356] "Generic (PLEG): container finished" podID="647187d5-0217-4a90-a5c7-900553dfd802" containerID="f1276e6dbc8f5e8239d27a27e19386542e2d9bc5bbf2e32f85c892377623ec5d" exitCode=0 Oct 30 18:19:32.088628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.088603 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" event={"ID":"647187d5-0217-4a90-a5c7-900553dfd802","Type":"ContainerDied","Data":"f1276e6dbc8f5e8239d27a27e19386542e2d9bc5bbf2e32f85c892377623ec5d"} Oct 30 18:19:32.199669 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.199601 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:32.200038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.200018 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:19:32.329005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.326854 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-socket" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:32.332268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.303004 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-socket\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.332478 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.332443 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647187d5-0217-4a90-a5c7-900553dfd802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647187d5-0217-4a90-a5c7-900553dfd802-istiod-ca-cert\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.332790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.332769 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/647187d5-0217-4a90-a5c7-900553dfd802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647187d5-0217-4a90-a5c7-900553dfd802-istio-podinfo\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.332998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.332928 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-kube-api-access-p2x76\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-kube-api-access-p2x76\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.333313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.333281 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-certs\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.333530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.333512 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-data\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.333725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.333708 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-credential-socket\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.333889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.333863 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-envoy\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.334078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.334061 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-istio-token\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-istio-token\") pod \"647187d5-0217-4a90-a5c7-900553dfd802\" (UID: \"647187d5-0217-4a90-a5c7-900553dfd802\") " Oct 30 18:19:32.334389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.334365 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-certs" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:32.334739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.334681 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647187d5-0217-4a90-a5c7-900553dfd802-istiod-ca-cert" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:32.335688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.335643 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:32.335826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.335811 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647187d5-0217-4a90-a5c7-900553dfd802-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:32.335978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.335963 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:32.340127 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.340099 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-istio-token" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:32.340447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.340431 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-data" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:32.340827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.340809 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-credential-socket" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:32.344185 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.342063 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/647187d5-0217-4a90-a5c7-900553dfd802-istio-podinfo" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:32.346689 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.346653 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-kube-api-access-p2x76" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "kube-api-access-p2x76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:32.367619 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.366961 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-envoy" pod "647187d5-0217-4a90-a5c7-900553dfd802" (UID: "647187d5-0217-4a90-a5c7-900553dfd802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:32.437500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.437464 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:32.437500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.437503 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:32.437775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.437521 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647187d5-0217-4a90-a5c7-900553dfd802-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:32.437775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.437534 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:32.437775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.437547 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647187d5-0217-4a90-a5c7-900553dfd802-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:32.437775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:32.437560 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2x76\" (UniqueName: \"kubernetes.io/projected/647187d5-0217-4a90-a5c7-900553dfd802-kube-api-access-p2x76\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:33.106413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:33.106328 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" event={"ID":"647187d5-0217-4a90-a5c7-900553dfd802","Type":"ContainerDied","Data":"c8a36d25568b1e061ebd0066ce1996ee49be848ef68c4c54d83514afcf8bceb3"} Oct 30 18:19:33.107416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:33.106383 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf" Oct 30 18:19:33.107416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:33.106465 5624 scope.go:122] "RemoveContainer" containerID="f1276e6dbc8f5e8239d27a27e19386542e2d9bc5bbf2e32f85c892377623ec5d" Oct 30 18:19:33.181334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:33.181256 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf"] Oct 30 18:19:33.195050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:33.194951 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f6a224e7-openshift-default-7d8f55f89f-25vvf"] Oct 30 18:19:33.800271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:33.800184 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647187d5-0217-4a90-a5c7-900553dfd802" path="/var/lib/kubelet/pods/647187d5-0217-4a90-a5c7-900553dfd802/volumes" Oct 30 18:19:34.039152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:34.039089 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:35.621865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:35.621831 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:37.232547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.232475 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn"] Oct 30 18:19:37.233424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233406 5624 cpu_manager.go:414] "containerMap: removing container" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" Oct 30 18:19:37.233424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233425 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" Oct 30 18:19:37.233512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233436 5624 cpu_manager.go:414] "containerMap: removing container" podUID="647187d5-0217-4a90-a5c7-900553dfd802" containerName="istio-proxy" Oct 30 18:19:37.233512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233441 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647187d5-0217-4a90-a5c7-900553dfd802" containerName="istio-proxy" Oct 30 18:19:37.233512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233488 5624 cpu_manager.go:414] "containerMap: removing container" podUID="f7340928-a2ba-4a8c-923e-26eb14aafb8d" containerName="maas-api" Oct 30 18:19:37.233512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233493 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7340928-a2ba-4a8c-923e-26eb14aafb8d" containerName="maas-api" Oct 30 18:19:37.233630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233595 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" Oct 30 18:19:37.233630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233604 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7340928-a2ba-4a8c-923e-26eb14aafb8d" containerName="maas-api" Oct 30 18:19:37.233630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.233614 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="647187d5-0217-4a90-a5c7-900553dfd802" containerName="istio-proxy" Oct 30 18:19:37.237105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.237086 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.255762 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.253692 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn"] Oct 30 18:19:37.261950 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.260051 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b0171017-serving-cert\"" Oct 30 18:19:37.320955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.315374 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:37.352359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.352326 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-openshift-service-ca\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.352531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.352503 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-metrics-certs\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.352588 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.352534 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-maas-api-tls\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.352825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.352794 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6r65\" (UniqueName: \"kubernetes.io/projected/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-kube-api-access-k6r65\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.392912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.392881 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:37.454278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.454248 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6r65\" (UniqueName: \"kubernetes.io/projected/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-kube-api-access-k6r65\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.454442 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.454349 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-openshift-service-ca\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.454492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.454444 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-metrics-certs\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.454492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.454464 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-maas-api-tls\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.455147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.455121 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-openshift-service-ca\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.456971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.456948 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-metrics-certs\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.457218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.457199 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-maas-api-tls\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.474117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.474089 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6r65\" (UniqueName: \"kubernetes.io/projected/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-kube-api-access-k6r65\") pod \"maas-api-e2e-ait-b0171017-67d9654f8-xdmnn\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.493865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.493783 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:37.556635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.554301 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:37.894923 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:19:37.894894 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b19c954_6f5d_46ef_a3fe_bbeeb372e9a9.slice/crio-2f2d66efdb7d77bd39fb3994285118ce66f81b4a6edb5e30a458f27443e6dc01 WatchSource:0}: Error finding container 2f2d66efdb7d77bd39fb3994285118ce66f81b4a6edb5e30a458f27443e6dc01: Status 404 returned error can't find the container with id 2f2d66efdb7d77bd39fb3994285118ce66f81b4a6edb5e30a458f27443e6dc01 Oct 30 18:19:37.896184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:37.896158 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn"] Oct 30 18:19:38.337878 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:38.337846 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" event={"ID":"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9","Type":"ContainerStarted","Data":"2f2d66efdb7d77bd39fb3994285118ce66f81b4a6edb5e30a458f27443e6dc01"} Oct 30 18:19:38.401338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:38.401302 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:39.324360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:39.324308 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:39.349012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:39.348981 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" event={"ID":"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9","Type":"ContainerStarted","Data":"0f5874608b6953966be005068ed630a7a8511b1dde8702aab7e0f00e586f0158"} Oct 30 18:19:39.349390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:39.349075 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:41.066531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:41.066504 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:41.282438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:41.282408 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:41.383108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:41.383074 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:43.665787 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:43.665747 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:45.365347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:45.365317 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:19:45.416719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:45.416657 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" podStartSLOduration=8.057954647 podStartE2EDuration="8.416636737s" totalImagesPullingTime="358.68209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:39.394635768 +0000 UTC m=+1073.783163886" watchObservedRunningTime="2026-10-30 18:19:45.416636737 +0000 UTC m=+1079.805164855" Oct 30 18:19:45.711657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:45.711568 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:46.757502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:46.757460 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:46.836618 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:46.836559 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:47.045001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.044896 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:47.585040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.578612 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:47.780850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.780819 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:47.921326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.921292 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld"] Oct 30 18:19:47.922451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.922422 5624 cpu_manager.go:414] "containerMap: removing container" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" Oct 30 18:19:47.922598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.922450 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" Oct 30 18:19:47.922746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.922730 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="23496535-1b6c-4a4d-8080-a06ce5a26b93" containerName="istio-proxy" Oct 30 18:19:47.932007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.928754 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:47.958252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.958219 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld"] Oct 30 18:19:47.962433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.962411 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-273105-openshift-default-dockercfg-p57lq\"" Oct 30 18:19:47.972840 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:47.972806 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:48.001776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.001745 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-envoy\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.001967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.001831 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-certs\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.002057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.002028 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-data\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.002146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.002128 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-credential-socket\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.002207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.002167 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85fg\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-kube-api-access-f85fg\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.002270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.002258 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf67015-168c-4270-828d-b4e3ff14f47a-istiod-ca-cert\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.002325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.002295 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-socket\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.002430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.002416 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-podinfo\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.002490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.002444 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-token\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.112921 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-certs\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.113078 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-data\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.113147 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-credential-socket\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.113175 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85fg\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-kube-api-access-f85fg\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.113245 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf67015-168c-4270-828d-b4e3ff14f47a-istiod-ca-cert\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.113279 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-socket\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.113360 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-podinfo\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.113388 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-token\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.114959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.113541 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-envoy\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.137183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.135539 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-data\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.137183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.135797 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-certs\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.137183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.136088 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-socket\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.137183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.136734 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf67015-168c-4270-828d-b4e3ff14f47a-istiod-ca-cert\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.151519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.137859 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-credential-socket\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.164361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.159614 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-podinfo\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.193353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.193240 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-envoy\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.227183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.227107 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-token\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.228069 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.228019 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85fg\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-kube-api-access-f85fg\") pod \"e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.268567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.255685 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:48.308304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.308233 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:48.726029 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:19:48.726002 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf67015_168c_4270_828d_b4e3ff14f47a.slice/crio-d0880042b87e2cddbead6cca838e9555e8868772569a8991dc0ccd52c6b63921 WatchSource:0}: Error finding container d0880042b87e2cddbead6cca838e9555e8868772569a8991dc0ccd52c6b63921: Status 404 returned error can't find the container with id d0880042b87e2cddbead6cca838e9555e8868772569a8991dc0ccd52c6b63921 Oct 30 18:19:48.728832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.728800 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:48.728918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.728865 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:48.728918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.728892 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:19:48.729819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:48.729790 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld"] Oct 30 18:19:49.413607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:49.413574 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:49.566677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:49.566640 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" event={"ID":"8cf67015-168c-4270-828d-b4e3ff14f47a","Type":"ContainerStarted","Data":"f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce"} Oct 30 18:19:49.566853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:49.566705 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" event={"ID":"8cf67015-168c-4270-828d-b4e3ff14f47a","Type":"ContainerStarted","Data":"d0880042b87e2cddbead6cca838e9555e8868772569a8991dc0ccd52c6b63921"} Oct 30 18:19:49.625803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:49.625730 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" podStartSLOduration=2.62571156 podStartE2EDuration="2.62571156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:49.622313324 +0000 UTC m=+1084.010841434" watchObservedRunningTime="2026-10-30 18:19:49.62571156 +0000 UTC m=+1084.014239677" Oct 30 18:19:49.921620 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:19:49.921583 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:19:50.256857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:50.256748 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:50.265090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:50.265064 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:50.576693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:50.576591 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:50.577816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:50.577794 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:19:50.801057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:50.801022 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57"] Oct 30 18:19:50.801300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:50.801254 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" podUID="cec76070-478d-41dd-b3f6-9cb678ac839c" containerName="maas-api" containerID="cri-o://415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07" gracePeriod=30 Oct 30 18:19:50.861095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:50.861067 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:51.300050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.298531 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:51.628082 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.514648 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls\") pod \"cec76070-478d-41dd-b3f6-9cb678ac839c\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " Oct 30 18:19:51.628082 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.514724 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec76070-478d-41dd-b3f6-9cb678ac839c-kube-api-access-zj955\" (UniqueName: \"kubernetes.io/projected/cec76070-478d-41dd-b3f6-9cb678ac839c-kube-api-access-zj955\") pod \"cec76070-478d-41dd-b3f6-9cb678ac839c\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " Oct 30 18:19:51.628082 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.514971 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-metrics-certs\") pod \"cec76070-478d-41dd-b3f6-9cb678ac839c\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " Oct 30 18:19:51.628082 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.515195 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec76070-478d-41dd-b3f6-9cb678ac839c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec76070-478d-41dd-b3f6-9cb678ac839c-openshift-service-ca\") pod \"cec76070-478d-41dd-b3f6-9cb678ac839c\" (UID: \"cec76070-478d-41dd-b3f6-9cb678ac839c\") " Oct 30 18:19:51.682524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.668480 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls" pod "cec76070-478d-41dd-b3f6-9cb678ac839c" (UID: "cec76070-478d-41dd-b3f6-9cb678ac839c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:51.682524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.668953 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec76070-478d-41dd-b3f6-9cb678ac839c-openshift-service-ca" pod "cec76070-478d-41dd-b3f6-9cb678ac839c" (UID: "cec76070-478d-41dd-b3f6-9cb678ac839c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:51.684582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.684538 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-metrics-certs" pod "cec76070-478d-41dd-b3f6-9cb678ac839c" (UID: "cec76070-478d-41dd-b3f6-9cb678ac839c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:51.685332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.685277 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec76070-478d-41dd-b3f6-9cb678ac839c-kube-api-access-zj955" pod "cec76070-478d-41dd-b3f6-9cb678ac839c" (UID: "cec76070-478d-41dd-b3f6-9cb678ac839c"). InnerVolumeSpecName "kube-api-access-zj955". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:51.758956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.735597 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec76070-478d-41dd-b3f6-9cb678ac839c-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:51.758956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.741022 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:51.758956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.741049 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj955\" (UniqueName: \"kubernetes.io/projected/cec76070-478d-41dd-b3f6-9cb678ac839c-kube-api-access-zj955\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:51.758956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.741078 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec76070-478d-41dd-b3f6-9cb678ac839c-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:19:51.758956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.754481 5624 generic.go:356] "Generic (PLEG): container finished" podID="cec76070-478d-41dd-b3f6-9cb678ac839c" containerID="415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07" exitCode=0 Oct 30 18:19:51.758956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.755681 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" Oct 30 18:19:51.766955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.765109 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" event={"ID":"cec76070-478d-41dd-b3f6-9cb678ac839c","Type":"ContainerDied","Data":"415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07"} Oct 30 18:19:51.766955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.765173 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57" event={"ID":"cec76070-478d-41dd-b3f6-9cb678ac839c","Type":"ContainerDied","Data":"1e8a5f51f4bc532e01943397c8ed70dce5b1880cba33a3fd61a92885256c56db"} Oct 30 18:19:51.766955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.765201 5624 scope.go:122] "RemoveContainer" containerID="415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07" Oct 30 18:19:51.818352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.818272 5624 scope.go:122] "RemoveContainer" containerID="415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07" Oct 30 18:19:51.818696 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:19:51.818674 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07\": container with ID starting with 415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07 not found: ID does not exist" containerID="415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07" Oct 30 18:19:51.818775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.818707 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07"} err="failed to get container status \"415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07\": rpc error: code = NotFound desc = could not find container \"415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07\": container with ID starting with 415e48026334c9262cc5777499a7de3f871c5493c2578738f8fe684338279b07 not found: ID does not exist" Oct 30 18:19:51.828173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.828148 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:51.839202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.839174 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57"] Oct 30 18:19:51.843422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.843397 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a90958fa-7b5567cf45-g2l57"] Oct 30 18:19:51.926434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:51.926354 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:52.036062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:52.036002 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:52.144359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:52.144281 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:52.246588 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:52.246503 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:52.378960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:52.371505 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:52.505015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:52.488374 5624 scope.go:122] "RemoveContainer" containerID="ce7b518a44175b72d592dee87adf51c6619c16d63b85340ba846f22df2e17543" Oct 30 18:19:53.496978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:53.496768 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:53.599704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:53.599671 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:53.795789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:53.795709 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec76070-478d-41dd-b3f6-9cb678ac839c" path="/var/lib/kubelet/pods/cec76070-478d-41dd-b3f6-9cb678ac839c/volumes" Oct 30 18:19:53.916530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:53.916489 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:54.020864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.020832 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:54.125018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.124987 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x"] Oct 30 18:19:54.130107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.130078 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cec76070-478d-41dd-b3f6-9cb678ac839c" containerName="maas-api" Oct 30 18:19:54.130284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.130116 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec76070-478d-41dd-b3f6-9cb678ac839c" containerName="maas-api" Oct 30 18:19:54.130355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.130294 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec76070-478d-41dd-b3f6-9cb678ac839c" containerName="maas-api" Oct 30 18:19:54.135672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.135644 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:54.153505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.153481 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-273105\"" Oct 30 18:19:54.153723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.153709 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-273105-dockercfg-4zjdw\"" Oct 30 18:19:54.167000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.160339 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x"] Oct 30 18:19:54.323206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.304268 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:54.741896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.740017 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzvq\" (UniqueName: \"kubernetes.io/projected/ee5e82a6-72ac-49e5-aebd-86fe992c0919-kube-api-access-rbzvq\") pod \"payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x\" (UID: \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:54.741896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.740362 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5e82a6-72ac-49e5-aebd-86fe992c0919-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x\" (UID: \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:54.760724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.760694 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm"] Oct 30 18:19:54.768475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.768449 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:54.774853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.774820 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm"] Oct 30 18:19:54.841332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.841282 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5e82a6-72ac-49e5-aebd-86fe992c0919-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x\" (UID: \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:54.841542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.841382 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bqj\" (UniqueName: \"kubernetes.io/projected/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-kube-api-access-v9bqj\") pod \"payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm\" (UID: \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:54.841542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.841469 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm\" (UID: \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:54.841706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.841670 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzvq\" (UniqueName: \"kubernetes.io/projected/ee5e82a6-72ac-49e5-aebd-86fe992c0919-kube-api-access-rbzvq\") pod \"payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x\" (UID: \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:54.842283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.842255 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5e82a6-72ac-49e5-aebd-86fe992c0919-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x\" (UID: \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:54.861660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.861619 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzvq\" (UniqueName: \"kubernetes.io/projected/ee5e82a6-72ac-49e5-aebd-86fe992c0919-kube-api-access-rbzvq\") pod \"payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x\" (UID: \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:54.941535 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.941495 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:54.943058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.943023 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bqj\" (UniqueName: \"kubernetes.io/projected/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-kube-api-access-v9bqj\") pod \"payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm\" (UID: \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:54.943152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.943101 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm\" (UID: \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:54.943948 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.943906 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm\" (UID: \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:54.963773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:54.963725 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bqj\" (UniqueName: \"kubernetes.io/projected/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-kube-api-access-v9bqj\") pod \"payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm\" (UID: \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:55.088056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.088004 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:55.274270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.274026 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x"] Oct 30 18:19:55.276671 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:19:55.276633 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5e82a6_72ac_49e5_aebd_86fe992c0919.slice/crio-e56d7088fd7656406b711b4ce3a0138bf97573cef1dc0ddda980ee8a058139e9 WatchSource:0}: Error finding container e56d7088fd7656406b711b4ce3a0138bf97573cef1dc0ddda980ee8a058139e9: Status 404 returned error can't find the container with id e56d7088fd7656406b711b4ce3a0138bf97573cef1dc0ddda980ee8a058139e9 Oct 30 18:19:55.423535 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:19:55.423501 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d01b1fb_3d3b_4831_8d03_ca892b7f93b5.slice/crio-b746a6f775c9899e1cd539ef75ece9ea77858c884481a1a4c4af18d3a1017d2f WatchSource:0}: Error finding container b746a6f775c9899e1cd539ef75ece9ea77858c884481a1a4c4af18d3a1017d2f: Status 404 returned error can't find the container with id b746a6f775c9899e1cd539ef75ece9ea77858c884481a1a4c4af18d3a1017d2f Oct 30 18:19:55.424420 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.424398 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm"] Oct 30 18:19:55.799260 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.799226 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" event={"ID":"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5","Type":"ContainerStarted","Data":"678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45"} Oct 30 18:19:55.799691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.799283 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" event={"ID":"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5","Type":"ContainerStarted","Data":"b746a6f775c9899e1cd539ef75ece9ea77858c884481a1a4c4af18d3a1017d2f"} Oct 30 18:19:55.799691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.799330 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:19:55.801223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.801197 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" event={"ID":"ee5e82a6-72ac-49e5-aebd-86fe992c0919","Type":"ContainerStarted","Data":"e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7"} Oct 30 18:19:55.801579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.801242 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" event={"ID":"ee5e82a6-72ac-49e5-aebd-86fe992c0919","Type":"ContainerStarted","Data":"e56d7088fd7656406b711b4ce3a0138bf97573cef1dc0ddda980ee8a058139e9"} Oct 30 18:19:55.801579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.801484 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:19:55.842849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.842805 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" podStartSLOduration=1.842790559 podStartE2EDuration="1.842790559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:55.839506607 +0000 UTC m=+1090.228034736" watchObservedRunningTime="2026-10-30 18:19:55.842790559 +0000 UTC m=+1090.231318678" Oct 30 18:19:55.850301 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.850271 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:55.882380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:55.882319 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" podStartSLOduration=1.8823000730000001 podStartE2EDuration="1.882300073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:55.879345123 +0000 UTC m=+1090.267873238" watchObservedRunningTime="2026-10-30 18:19:55.882300073 +0000 UTC m=+1090.270828192" Oct 30 18:19:57.432085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:57.432004 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:59.354780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:19:59.347847 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:00.925253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:00.925194 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:02.642264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:02.642225 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:04.381615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:04.381579 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:06.435629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:06.435593 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:06.533204 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:06.533152 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:06.823006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:06.822372 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:20:06.826951 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:06.826788 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:20:08.341959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:08.341890 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:09.246989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:09.246609 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:09.258957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:09.257014 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:09.363959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:09.363665 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:09.414957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:09.411071 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:09.684957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:09.684229 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:09.876959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:09.876437 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:10.255958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.254915 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:10.255958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.255555 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:20:10.275972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.262775 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:10.275972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.263336 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:20:10.310114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.309158 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:10.353962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.338143 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:10.380998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.380043 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:10.380998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.380497 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:20:10.482005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.412894 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:10.482005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.413305 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:20:10.482005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.448839 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:10.907996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.895472 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:10.966169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:10.932846 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd"] Oct 30 18:20:11.263044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.227586 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:11.294811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.266871 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:11.294811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.292116 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:11.483285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.483238 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:11.491073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.491028 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:11.548075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.538761 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:11.548075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.538861 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:11.551956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.548446 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:11.628995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.614197 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:20:11.709317 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.709274 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:20:11.920985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.913916 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:20:12.012205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:11.766535 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:20:12.012205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.007029 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:20:12.114960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.035299 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:12.121961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.064429 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.132791 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.113764 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1085804-d290-4dfd-8b33-bbbbc3c28009-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.133297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.133270 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.136962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.133561 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.167249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.167211 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72h9\" (UniqueName: \"kubernetes.io/projected/a1085804-d290-4dfd-8b33-bbbbc3c28009-kube-api-access-x72h9\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.167713 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.158915 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-273105-serving-cert\"" Oct 30 18:20:12.301980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.298148 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:20:12.301980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.298840 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:12.327972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.302824 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.335009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.334966 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72h9\" (UniqueName: \"kubernetes.io/projected/a1085804-d290-4dfd-8b33-bbbbc3c28009-kube-api-access-x72h9\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.335269 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.335246 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1085804-d290-4dfd-8b33-bbbbc3c28009-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.335732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.335710 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.350405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.350372 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1085804-d290-4dfd-8b33-bbbbc3c28009-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.526994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.526901 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.621972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.552307 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:12.622292 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.622260 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:12.646960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.570178 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:12.646960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.595519 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:12.646960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.569524 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.694372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.694337 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:20:12.694896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.694875 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:20:12.695072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.695056 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd"] Oct 30 18:20:12.716573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.716536 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:12.786641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.758711 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72h9\" (UniqueName: \"kubernetes.io/projected/a1085804-d290-4dfd-8b33-bbbbc3c28009-kube-api-access-x72h9\") pod \"maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.789703 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.789673 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:12.890972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.882265 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:12.997366 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:12.997324 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:13.079164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.065140 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:13.097958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.091370 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:20:13.309492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.309449 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:13.312422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.310804 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:13.627640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.627581 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:20:13.663732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.661403 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:20:13.663732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.661781 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:20:13.723873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.723834 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:13.892665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.892576 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:14.000967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:13.999024 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:20:14.164497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.164403 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:14.374960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.356841 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:14.374960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.357701 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:14.596470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.596357 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:14.713975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.707380 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:20:14.713975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.707691 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:20:14.743977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.743911 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:14.766136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.749966 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:14.823150 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.822059 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:14.854209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.853137 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:20:14.856694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.856659 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd"] Oct 30 18:20:15.018959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:14.994634 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:20:15.018959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.001300 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:15.068583 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.068557 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:20:15.167238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.165055 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:15.183746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.183718 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:20:15.238955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.238622 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:20:15.527059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.485553 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 18:20:15.685019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.684986 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" event={"ID":"a1085804-d290-4dfd-8b33-bbbbc3c28009","Type":"ContainerStarted","Data":"3bf84416fd7ee8c15f5bf783f1db38a1bf08c63f0ff5e2a5670a0ceee889cde0"} Oct 30 18:20:15.725956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.708948 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:15.774561 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.756729 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:15.783196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.783172 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:20:15.977546 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:15.975257 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:16.023242 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:16.023198 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:16.419224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:16.366022 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:20:16.419224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:16.414447 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:16.446960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:16.443842 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:20:16.757053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:16.755710 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" event={"ID":"a1085804-d290-4dfd-8b33-bbbbc3c28009","Type":"ContainerStarted","Data":"23f5bf2cca0fc069817af4d1bcda2e83d4817735587037f3f9987debdb89df8c"} Oct 30 18:20:16.757053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:16.756058 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:16.807112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:16.802801 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" podStartSLOduration=5.9891883759999995 podStartE2EDuration="6.802782322s" totalImagesPullingTime="813.593946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:16.796113458 +0000 UTC m=+1111.184641584" watchObservedRunningTime="2026-10-30 18:20:16.802782322 +0000 UTC m=+1111.191310439" Oct 30 18:20:17.417982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:17.410183 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn"] Oct 30 18:20:17.417982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:17.410424 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" podUID="3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" containerName="maas-api" containerID="cri-o://0f5874608b6953966be005068ed630a7a8511b1dde8702aab7e0f00e586f0158" gracePeriod=30 Oct 30 18:20:17.464832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:17.464391 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:18.028070 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.007323 5624 generic.go:356] "Generic (PLEG): container finished" podID="3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" containerID="0f5874608b6953966be005068ed630a7a8511b1dde8702aab7e0f00e586f0158" exitCode=0 Oct 30 18:20:18.028070 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.026726 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" event={"ID":"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9","Type":"ContainerDied","Data":"0f5874608b6953966be005068ed630a7a8511b1dde8702aab7e0f00e586f0158"} Oct 30 18:20:18.253563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.238509 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:18.255500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.255047 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:20:18.316399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.307331 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-kube-api-access-k6r65\" (UniqueName: \"kubernetes.io/projected/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-kube-api-access-k6r65\") pod \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " Oct 30 18:20:18.316399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.307387 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-metrics-certs\") pod \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " Oct 30 18:20:18.316399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.307535 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-openshift-service-ca\") pod \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " Oct 30 18:20:18.316399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.307682 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-maas-api-tls\") pod \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\" (UID: \"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9\") " Oct 30 18:20:18.324018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.317124 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-openshift-service-ca" pod "3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" (UID: "3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:18.324018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.318238 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:18.324018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.321652 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:18.367202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.366365 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-kube-api-access-k6r65" pod "3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" (UID: "3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9"). InnerVolumeSpecName "kube-api-access-k6r65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:18.367202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.366488 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-metrics-certs" pod "3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" (UID: "3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:18.368978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.368057 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-maas-api-tls" pod "3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" (UID: "3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:18.419302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.419264 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:18.420213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.420176 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6r65\" (UniqueName: \"kubernetes.io/projected/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-kube-api-access-k6r65\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:18.420324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.420218 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:18.420324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.420233 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:18.994241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.992416 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr"] Oct 30 18:20:18.994241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:18.992741 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" podUID="a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" containerName="istio-proxy" containerID="cri-o://b4096edb54b80e0fbb40604e9788d453d7117a454bdddd89f6438c6449675f97" gracePeriod=30 Oct 30 18:20:19.009038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:19.007221 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:19.027813 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:19.027729 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" event={"ID":"3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9","Type":"ContainerDied","Data":"2f2d66efdb7d77bd39fb3994285118ce66f81b4a6edb5e30a458f27443e6dc01"} Oct 30 18:20:19.027813 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:19.027794 5624 scope.go:122] "RemoveContainer" containerID="0f5874608b6953966be005068ed630a7a8511b1dde8702aab7e0f00e586f0158" Oct 30 18:20:19.028078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:19.027991 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn" Oct 30 18:20:19.329051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:19.329016 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn"] Oct 30 18:20:19.376033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:19.367732 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b0171017-67d9654f8-xdmnn"] Oct 30 18:20:19.816005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:19.815704 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:19.824972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:19.823892 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" path="/var/lib/kubelet/pods/3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9/volumes" Oct 30 18:20:19.940677 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:20:19.939712 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:20:20.244989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.240848 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft"] Oct 30 18:20:20.244989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.243867 5624 cpu_manager.go:414] "containerMap: removing container" podUID="3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" containerName="maas-api" Oct 30 18:20:20.244989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.243897 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" containerName="maas-api" Oct 30 18:20:20.244989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.244124 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b19c954-6f5d-46ef-a3fe-bbeeb372e9a9" containerName="maas-api" Oct 30 18:20:20.258492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.258245 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.302772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.293714 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ae3373-openshift-default-dockercfg-kcj2c\"" Oct 30 18:20:20.302772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.300141 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:20.319976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.319923 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft"] Oct 30 18:20:20.333519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.332571 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rbf\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-kube-api-access-m7rbf\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.333519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.332795 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.333519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.333112 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.333519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.333283 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.333519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.333363 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.333519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.333432 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.333519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.333509 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.334305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.333536 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e369dc-6ca5-45f2-9c24-00d8a7421900-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.334305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.333610 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.432445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.432411 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:20.437286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437245 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.437473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437340 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.437473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437367 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.437473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437389 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.437473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437425 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.437473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437443 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e369dc-6ca5-45f2-9c24-00d8a7421900-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.437473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437475 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.437788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437530 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rbf\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-kube-api-access-m7rbf\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.437788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.437618 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.438197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.438169 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.438305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.438235 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.438901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.438869 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e369dc-6ca5-45f2-9c24-00d8a7421900-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.438901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.438883 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.439129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.439046 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.455459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.455321 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.455824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.455794 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.459227 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.459164 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rbf\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-kube-api-access-m7rbf\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.459944 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.459901 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:20.537543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.537397 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:20.704102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:20.697317 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:21.303810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:21.303674 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:21.479808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:21.479774 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft"] Oct 30 18:20:21.500383 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:20:21.489024 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e369dc_6ca5_45f2_9c24_00d8a7421900.slice/crio-85638057edf6383124e3ef0183fd01a7001032c3e28f42b318e7f90a14996697 WatchSource:0}: Error finding container 85638057edf6383124e3ef0183fd01a7001032c3e28f42b318e7f90a14996697: Status 404 returned error can't find the container with id 85638057edf6383124e3ef0183fd01a7001032c3e28f42b318e7f90a14996697 Oct 30 18:20:21.500383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:21.492733 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:21.500383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:21.492817 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:21.500383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:21.492863 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:22.070414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:22.070378 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" event={"ID":"47e369dc-6ca5-45f2-9c24-00d8a7421900","Type":"ContainerStarted","Data":"3e444475098266a85924106c0ef783198bb61b7068a28747ef16e3c1ca63d06e"} Oct 30 18:20:22.070597 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:22.070447 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" event={"ID":"47e369dc-6ca5-45f2-9c24-00d8a7421900","Type":"ContainerStarted","Data":"85638057edf6383124e3ef0183fd01a7001032c3e28f42b318e7f90a14996697"} Oct 30 18:20:22.127581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:22.127508 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" podStartSLOduration=2.127487581 podStartE2EDuration="2.127487581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:22.121432959 +0000 UTC m=+1116.509961078" watchObservedRunningTime="2026-10-30 18:20:22.127487581 +0000 UTC m=+1116.516015699" Oct 30 18:20:22.130672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:22.130636 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:22.750389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:22.750358 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:22.756416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:22.756390 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:23.771593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:23.771553 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:20:23.781957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:23.776198 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:23.781957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:23.777377 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:20:24.536047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:24.536016 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:24.637678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:24.637636 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:24.882414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:24.882357 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" podUID="0a09133a-00c6-4632-ba5e-b859f8da282e" containerName="server" probeResult="failure" output="Get \"https://10.134.0.37:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:25.630580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:25.630544 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:26.925434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:26.925400 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2"] Oct 30 18:20:27.040316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.040266 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:27.040503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.040344 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:27.060973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.060924 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:27.134306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.134274 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2"] Oct 30 18:20:27.134510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.134367 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.159595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.159562 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:27.227604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.227523 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jslw\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-kube-api-access-9jslw\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.227752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.227708 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.227855 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.227837 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.227904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.227885 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.228131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.228115 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10cb470-5e0a-4def-806d-0e08450f10a2-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.228545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.228203 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.228660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.228575 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.228660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.228629 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.228824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.228661 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.331380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.331328 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.331576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.331460 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jslw\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-kube-api-access-9jslw\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.331649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.331605 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.331802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.331743 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.331866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.331796 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.332079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.332032 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10cb470-5e0a-4def-806d-0e08450f10a2-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.332206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.332157 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.332321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.332249 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.332321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.332310 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.333060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.333009 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.333333 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.333270 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.333835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.333810 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.334146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.334093 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.342896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.342191 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10cb470-5e0a-4def-806d-0e08450f10a2-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.360026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.357294 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:27.364110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.363803 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.365040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.364982 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.441979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.430648 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jslw\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-kube-api-access-9jslw\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.441979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.431157 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:27.555048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:27.554100 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:28.465812 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:20:28.465780 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10cb470_5e0a_4def_806d_0e08450f10a2.slice/crio-88e5df1dd892012347f91c97752ede7a417348ca9857685695970340a20b745a WatchSource:0}: Error finding container 88e5df1dd892012347f91c97752ede7a417348ca9857685695970340a20b745a: Status 404 returned error can't find the container with id 88e5df1dd892012347f91c97752ede7a417348ca9857685695970340a20b745a Oct 30 18:20:28.468101 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:28.468073 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2"] Oct 30 18:20:28.469525 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:28.469485 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:28.469639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:28.469569 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:28.469639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:28.469608 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:28.707058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:28.707022 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" event={"ID":"a10cb470-5e0a-4def-806d-0e08450f10a2","Type":"ContainerStarted","Data":"b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882"} Oct 30 18:20:28.707245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:28.707087 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" event={"ID":"a10cb470-5e0a-4def-806d-0e08450f10a2","Type":"ContainerStarted","Data":"88e5df1dd892012347f91c97752ede7a417348ca9857685695970340a20b745a"} Oct 30 18:20:28.754304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:28.754198 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" podStartSLOduration=2.7541834400000003 podStartE2EDuration="2.75418344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:28.749798076 +0000 UTC m=+1123.138326198" watchObservedRunningTime="2026-10-30 18:20:28.75418344 +0000 UTC m=+1123.142711557" Oct 30 18:20:29.428246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:29.428204 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:29.562351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:29.559299 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:29.571960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:29.563047 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:20:29.571960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:29.563111 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:20:30.555408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:30.555368 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:20:30.555664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:30.555429 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:20:31.292451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:31.291482 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:31.453987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:31.420149 5624 patch_prober.go:29] interesting pod/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:31.453987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:31.420209 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" podUID="a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:31.555568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:31.555488 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:20:31.555568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:31.555541 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:20:32.555817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:32.555778 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:20:32.556350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:32.555834 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:20:32.612830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:32.612793 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:33.122443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:33.122400 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:33.556263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:33.556189 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:20:33.556647 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:33.556267 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:20:34.196344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:34.196215 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:34.556553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:34.556506 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 18:20:34.557063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:34.556581 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 18:20:34.666902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:34.666856 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:35.107756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.107717 5624 generic.go:356] "Generic (PLEG): container finished" podID="a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" containerID="b4096edb54b80e0fbb40604e9788d453d7117a454bdddd89f6438c6449675f97" exitCode=0 Oct 30 18:20:35.108062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.108032 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" event={"ID":"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b","Type":"ContainerDied","Data":"b4096edb54b80e0fbb40604e9788d453d7117a454bdddd89f6438c6449675f97"} Oct 30 18:20:35.108167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.108103 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" event={"ID":"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b","Type":"ContainerDied","Data":"aa0873374b75e5992ad2427581267924a79df5bc2ad996f9cfa0de2718f30ff7"} Oct 30 18:20:35.108167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.108128 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa0873374b75e5992ad2427581267924a79df5bc2ad996f9cfa0de2718f30ff7" Oct 30 18:20:35.138994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.133738 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:35.162784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.162357 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.206997 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istiod-ca-cert\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.207121 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-kube-api-access-fwr76\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-kube-api-access-fwr76\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.208272 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istiod-ca-cert" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.208332 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-data\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.208441 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-podinfo\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.208532 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-envoy\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.208661 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-token\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-token\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.208701 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-socket\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.208805 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-certs\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.208848 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-credential-socket\") pod \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\" (UID: \"a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b\") " Oct 30 18:20:35.209820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.209458 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.210849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.210756 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-socket" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:35.211422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.211124 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-data" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:35.211422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.211242 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-certs" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:35.211612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.211590 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-credential-socket" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:35.228091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.228066 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-token" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:35.228915 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.228892 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-podinfo" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:20:35.235866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.234038 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-envoy" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:20:35.235866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.235704 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:35.273107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.269535 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-kube-api-access-fwr76" pod "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" (UID: "a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b"). InnerVolumeSpecName "kube-api-access-fwr76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:35.328700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.328574 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.328700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.328611 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.328700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.328624 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.328700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.328640 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.328700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.328655 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwr76\" (UniqueName: \"kubernetes.io/projected/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-kube-api-access-fwr76\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.328700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.328668 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.328700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.328682 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.328700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.328695 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:20:35.560702 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.560671 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:35.561161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.561140 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:35.561898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:35.561879 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:20:36.120531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.120430 5624 generic.go:356] "Generic (PLEG): container finished" podID="d50df807-dc42-412b-8264-2ddc87c48176" containerID="e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325" exitCode=0 Oct 30 18:20:36.120531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.120481 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" event={"ID":"d50df807-dc42-412b-8264-2ddc87c48176","Type":"ContainerDied","Data":"e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325"} Oct 30 18:20:36.121108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.121083 5624 scope.go:122] "RemoveContainer" containerID="e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325" Oct 30 18:20:36.121260 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.121239 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr" Oct 30 18:20:36.122877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.122851 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:36.122973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.122915 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:36.122973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.122964 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:20:36.207905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.207876 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr"] Oct 30 18:20:36.212540 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.212514 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b0171017-openshift-default-56f99f8d7-4k5lr"] Oct 30 18:20:36.955275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:36.955240 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:37.178403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:37.178368 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" event={"ID":"d50df807-dc42-412b-8264-2ddc87c48176","Type":"ContainerStarted","Data":"a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c"} Oct 30 18:20:37.749138 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:37.749106 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:20:37.750219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:37.750190 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:20:37.750342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:37.750250 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 18:20:37.817336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:37.817307 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" path="/var/lib/kubelet/pods/a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b/volumes" Oct 30 18:20:38.720491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:38.720448 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:38.749534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:38.749499 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:20:38.749675 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:38.749579 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 18:20:38.892802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:38.892754 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:39.003593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:39.003499 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:39.748842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:39.748799 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:20:39.749281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:39.748882 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 18:20:40.540233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.540194 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:40.668557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.668523 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:40.758375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.749690 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:20:40.758375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.749756 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 18:20:40.776723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.776672 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:40.874721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.874692 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw"] Oct 30 18:20:40.875479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.875464 5624 cpu_manager.go:414] "containerMap: removing container" podUID="a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" containerName="istio-proxy" Oct 30 18:20:40.875567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.875480 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" containerName="istio-proxy" Oct 30 18:20:40.876111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.875604 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8eafb2b-86dc-4a79-8ff6-948cfcf01b5b" containerName="istio-proxy" Oct 30 18:20:40.879898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.879877 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:40.929986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.929751 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fb0abbf7-dockercfg-z4rks\"" Oct 30 18:20:40.929986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.929847 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fb0abbf7\"" Oct 30 18:20:40.934975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.934928 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw"] Oct 30 18:20:40.958963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.957512 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29"] Oct 30 18:20:40.974393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.968008 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:41.004406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:40.999047 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:20:41.058953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.055372 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk6f9\" (UniqueName: \"kubernetes.io/projected/f5b2832b-135a-4f42-8864-15f49be4c6a6-kube-api-access-tk6f9\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29\" (UID: \"f5b2832b-135a-4f42-8864-15f49be4c6a6\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:41.058953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.055619 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtd68\" (UniqueName: \"kubernetes.io/projected/01d378e9-9da2-479b-8856-200a64f66c55-kube-api-access-dtd68\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw\" (UID: \"01d378e9-9da2-479b-8856-200a64f66c55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:41.058953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.055700 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b2832b-135a-4f42-8864-15f49be4c6a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29\" (UID: \"f5b2832b-135a-4f42-8864-15f49be4c6a6\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:41.058953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.055788 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d378e9-9da2-479b-8856-200a64f66c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw\" (UID: \"01d378e9-9da2-479b-8856-200a64f66c55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:41.165219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.164897 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtd68\" (UniqueName: \"kubernetes.io/projected/01d378e9-9da2-479b-8856-200a64f66c55-kube-api-access-dtd68\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw\" (UID: \"01d378e9-9da2-479b-8856-200a64f66c55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:41.165219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.164986 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b2832b-135a-4f42-8864-15f49be4c6a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29\" (UID: \"f5b2832b-135a-4f42-8864-15f49be4c6a6\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:41.165219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.165064 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d378e9-9da2-479b-8856-200a64f66c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw\" (UID: \"01d378e9-9da2-479b-8856-200a64f66c55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:41.165481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.165264 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk6f9\" (UniqueName: \"kubernetes.io/projected/f5b2832b-135a-4f42-8864-15f49be4c6a6-kube-api-access-tk6f9\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29\" (UID: \"f5b2832b-135a-4f42-8864-15f49be4c6a6\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:41.194958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.192107 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d378e9-9da2-479b-8856-200a64f66c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw\" (UID: \"01d378e9-9da2-479b-8856-200a64f66c55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:41.327963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.325321 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:41.367977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.354161 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29"] Oct 30 18:20:41.435007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.416654 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b2832b-135a-4f42-8864-15f49be4c6a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29\" (UID: \"f5b2832b-135a-4f42-8864-15f49be4c6a6\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:41.435007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.421830 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk6f9\" (UniqueName: \"kubernetes.io/projected/f5b2832b-135a-4f42-8864-15f49be4c6a6-kube-api-access-tk6f9\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29\" (UID: \"f5b2832b-135a-4f42-8864-15f49be4c6a6\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:41.452091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.450275 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtd68\" (UniqueName: \"kubernetes.io/projected/01d378e9-9da2-479b-8856-200a64f66c55-kube-api-access-dtd68\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw\" (UID: \"01d378e9-9da2-479b-8856-200a64f66c55\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:41.543407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.543373 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:41.552885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.552857 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:41.750538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.749969 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:20:41.750538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.750042 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 18:20:41.933174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.933107 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29"] Oct 30 18:20:41.937060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:41.937037 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw"] Oct 30 18:20:42.346623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.346534 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" event={"ID":"f5b2832b-135a-4f42-8864-15f49be4c6a6","Type":"ContainerStarted","Data":"312c551f0b54573928369e8cbfa01d9ed6ede085630b13ab90ef3099609a7778"} Oct 30 18:20:42.346623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.346584 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" event={"ID":"f5b2832b-135a-4f42-8864-15f49be4c6a6","Type":"ContainerStarted","Data":"f4ef12051c1377544f671f2847930ce3186428499e91baed43b42fffcff2b5fb"} Oct 30 18:20:42.347704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.347679 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:42.349603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.349572 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" event={"ID":"01d378e9-9da2-479b-8856-200a64f66c55","Type":"ContainerStarted","Data":"b65fba361799f6bdda7a0ddf7df18f305a0289bb95461095c24bab70632802d8"} Oct 30 18:20:42.349719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.349631 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" event={"ID":"01d378e9-9da2-479b-8856-200a64f66c55","Type":"ContainerStarted","Data":"0d10994020760797631469cd7f5e0cf95ca2b7a7161afee3da160e355aac70b2"} Oct 30 18:20:42.350470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.350446 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:42.380690 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.380668 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:42.397819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.396369 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" podStartSLOduration=2.396356408 podStartE2EDuration="2.396356408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:42.391338529 +0000 UTC m=+1136.779866645" watchObservedRunningTime="2026-10-30 18:20:42.396356408 +0000 UTC m=+1136.784884540" Oct 30 18:20:42.437167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.437119 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" podStartSLOduration=2.437104823 podStartE2EDuration="2.437104823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:42.431864003 +0000 UTC m=+1136.820392120" watchObservedRunningTime="2026-10-30 18:20:42.437104823 +0000 UTC m=+1136.825632940" Oct 30 18:20:42.458884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.458848 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:42.558864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.558775 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:42.764807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.764693 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:20:42.767463 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:42.767438 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:20:44.299060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:44.299026 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:45.916126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:45.916076 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:47.549608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:47.549568 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:49.076260 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:49.076227 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:49.938865 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:20:49.938829 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:20:50.711690 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:50.711623 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:51.371272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:51.371239 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:20:52.907403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:52.904079 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:54.426298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:54.425095 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:20:54.431107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:54.428080 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:20:54.994491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:54.994457 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:55.292147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.292037 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:55.519961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.488362 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:55.614275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.614174 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:55.722054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.722015 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:20:55.724890 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.724860 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:20:55.726008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.725359 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:20:55.746524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.746498 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:20:55.746734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.746711 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:55.817215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:55.817175 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:56.044161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:56.044099 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:56.240503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:56.240456 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:56.357269 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:56.357176 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:20:56.357992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:56.357966 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:20:56.410674 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:56.410222 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:20:56.786312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:56.786267 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:58.310133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.310098 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd"] Oct 30 18:20:58.317121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.317083 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.340280 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.340236 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd"] Oct 30 18:20:58.369089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.369061 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fb0abbf7-serving-cert\"" Oct 30 18:20:58.371288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.371257 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60543fc7-02a7-403a-bb27-bd46cd63591a-openshift-service-ca\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.371693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.371670 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.372064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.372045 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnn8j\" (UniqueName: \"kubernetes.io/projected/60543fc7-02a7-403a-bb27-bd46cd63591a-kube-api-access-xnn8j\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.372314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.372296 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-metrics-certs\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.465177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.465136 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:58.483350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.483312 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnn8j\" (UniqueName: \"kubernetes.io/projected/60543fc7-02a7-403a-bb27-bd46cd63591a-kube-api-access-xnn8j\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.484852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.484822 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-metrics-certs\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.485170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.485150 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60543fc7-02a7-403a-bb27-bd46cd63591a-openshift-service-ca\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.485406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.485388 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.499960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.486352 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60543fc7-02a7-403a-bb27-bd46cd63591a-openshift-service-ca\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.499960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.488614 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-metrics-certs\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.594704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.584698 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.597170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.597019 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnn8j\" (UniqueName: \"kubernetes.io/projected/60543fc7-02a7-403a-bb27-bd46cd63591a-kube-api-access-xnn8j\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:58.618893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.618853 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:58.685595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:58.685311 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:20:59.211664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:59.210273 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd"] Oct 30 18:20:59.213214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:20:59.213156 5624 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:21:00.213467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:00.213426 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" event={"ID":"60543fc7-02a7-403a-bb27-bd46cd63591a","Type":"ContainerStarted","Data":"38441a79b7ace0de8d22eb18626f1082a6f1554e678a1d8774c44e02fa59428e"} Oct 30 18:21:00.213970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:00.213495 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" event={"ID":"60543fc7-02a7-403a-bb27-bd46cd63591a","Type":"ContainerStarted","Data":"eefaf28a019c143396792100e7790524f920e64069a2e27850051cb9b6db7462"} Oct 30 18:21:00.213970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:00.213528 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:21:00.255636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:00.255578 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" podStartSLOduration=1.914494227 podStartE2EDuration="2.25556436s" totalImagesPullingTime="341.070133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:00.25327214 +0000 UTC m=+1154.641800258" watchObservedRunningTime="2026-10-30 18:21:00.25556436 +0000 UTC m=+1154.644092474" Oct 30 18:21:00.476711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:00.476626 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:02.332580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:02.332542 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:04.107094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:04.107034 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:05.958604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:05.958559 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:06.231451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:06.231372 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:21:07.722733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:07.722698 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:09.273640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:09.273611 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:10.261037 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:10.261002 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:10.740720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:10.740687 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:10.934418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:10.934384 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:11.655763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.655726 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt"] Oct 30 18:21:11.662264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.662240 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.698203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.698163 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e5d168-openshift-default-dockercfg-hlbp7\"" Oct 30 18:21:11.705822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.705791 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt"] Oct 30 18:21:11.759757 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.759719 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.759884 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626ce00-6a12-48e5-a761-c20623fd714a-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.760149 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.760184 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfllm\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-kube-api-access-tfllm\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.760316 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.760408 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.760440 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.760521 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.760609 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626ce00-6a12-48e5-a761-c20623fd714a-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.760851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.760720 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.864829 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626ce00-6a12-48e5-a761-c20623fd714a-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.865105 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.865194 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfllm\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-kube-api-access-tfllm\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.865439 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.865559 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.865592 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.865719 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.865855 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626ce00-6a12-48e5-a761-c20623fd714a-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.865890 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.866530 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.867773 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626ce00-6a12-48e5-a761-c20623fd714a-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.868958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.868307 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.871459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.871128 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.871459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.871261 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:11.871676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.871552 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.871995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.871967 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.875816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.875796 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626ce00-6a12-48e5-a761-c20623fd714a-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.889031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.889008 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfllm\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-kube-api-access-tfllm\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.892148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.891361 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:11.978782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:11.978701 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:12.319802 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:21:12.319771 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8626ce00_6a12_48e5_a761_c20623fd714a.slice/crio-cb48ef7807c0a29a735af15b17f4f94028e8b0b909a4aacc171ea96d968bc3c8 WatchSource:0}: Error finding container cb48ef7807c0a29a735af15b17f4f94028e8b0b909a4aacc171ea96d968bc3c8: Status 404 returned error can't find the container with id cb48ef7807c0a29a735af15b17f4f94028e8b0b909a4aacc171ea96d968bc3c8 Oct 30 18:21:12.322320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:12.322289 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:21:12.322408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:12.322354 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:21:12.322408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:12.322390 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:21:12.323823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:12.323793 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt"] Oct 30 18:21:12.378445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:12.378414 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" event={"ID":"8626ce00-6a12-48e5-a761-c20623fd714a","Type":"ContainerStarted","Data":"cb48ef7807c0a29a735af15b17f4f94028e8b0b909a4aacc171ea96d968bc3c8"} Oct 30 18:21:12.808738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:12.808700 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:13.388789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:13.388742 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" event={"ID":"8626ce00-6a12-48e5-a761-c20623fd714a","Type":"ContainerStarted","Data":"6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25"} Oct 30 18:21:13.444512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:13.444461 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podStartSLOduration=2.444443569 podStartE2EDuration="2.444443569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:13.443682113 +0000 UTC m=+1167.832210231" watchObservedRunningTime="2026-10-30 18:21:13.444443569 +0000 UTC m=+1167.832971683" Oct 30 18:21:13.979556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:13.979501 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:13.986982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:13.986952 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:14.399648 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:14.399619 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:14.400363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:14.400341 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:21:14.654434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:14.654342 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:16.337527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:16.337488 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:18.047922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:18.047891 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:19.749515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:19.749442 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:19.970331 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:21:19.970279 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:21:21.501956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:21.496461 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:23.216755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:23.216650 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:24.813322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:24.813290 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:25.861189 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:25.861154 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:25.959599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:25.959565 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:26.672811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:26.672775 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:27.535296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:27.535251 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:27.634414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:27.634363 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:28.565823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:28.565783 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:30.228858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:30.228819 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:32.490802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:32.490773 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:34.217376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:34.217345 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:34.308973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:34.308811 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:35.994962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:35.991504 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:36.288475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:36.288398 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:36.739101 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:36.737210 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:37.922609 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:37.922549 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:39.541979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:39.541919 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:39.818209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:39.818108 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:21:39.818209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:39.818181 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:21:40.929353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:40.929313 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:41.029286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:41.029257 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:41.155122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:41.155087 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:41.441612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:41.441580 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:42.844730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:42.844691 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:44.550352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:44.550314 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:46.116563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:46.116520 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:46.197893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:46.197865 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:46.299496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:46.299397 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:47.252288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:47.252246 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:47.369655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:47.369607 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:48.433407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:48.433374 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:49.918389 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:21:49.918353 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:21:53.041309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:53.041271 5624 generic.go:356] "Generic (PLEG): container finished" podID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerID="3e444475098266a85924106c0ef783198bb61b7068a28747ef16e3c1ca63d06e" exitCode=0 Oct 30 18:21:53.041858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:53.041343 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" event={"ID":"47e369dc-6ca5-45f2-9c24-00d8a7421900","Type":"ContainerDied","Data":"3e444475098266a85924106c0ef783198bb61b7068a28747ef16e3c1ca63d06e"} Oct 30 18:21:53.041858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:53.041753 5624 scope.go:122] "RemoveContainer" containerID="3e444475098266a85924106c0ef783198bb61b7068a28747ef16e3c1ca63d06e" Oct 30 18:21:53.043369 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:53.043341 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:21:53.043466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:53.043409 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:21:53.043466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:53.043440 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:21:53.776679 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:53.776644 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:21:54.052534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:54.052453 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" event={"ID":"47e369dc-6ca5-45f2-9c24-00d8a7421900","Type":"ContainerStarted","Data":"45da120f5ee93f4c9ad309b6ce3d7bebce1bb59f4f28b5b3861b836f8d9a1653"} Oct 30 18:21:54.750709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:54.750672 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:21:54.751846 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:54.751822 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:21:54.751993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:54.751874 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 18:21:55.750447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:55.750415 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:21:55.750850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:55.750473 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 18:21:56.750748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:56.750711 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:21:56.751429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:56.751395 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 18:21:57.240700 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:21:57.240654 5624 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-133-47.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:21:57.751235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:57.751201 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:21:57.751701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:57.751256 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 18:21:58.750857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:58.750824 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 18:21:58.751043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:58.750877 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 18:21:59.755515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:59.755485 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:21:59.755981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:59.755750 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:21:59.756755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:21:59.756732 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:22:07.241547 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:07.241498 5624 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-133-47.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:15.441075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:15.441039 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:22:15.441568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:15.441116 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:17.537431 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:17.537378 5624 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-133-47.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:22:20.909832 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:20.909789 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:22:27.722401 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:27.722358 5624 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-133-47.ec2.internal?timeout=10s\": context deadline exceeded" Oct 30 18:22:27.821913 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:27.821867 5624 kubelet_node_status.go:524] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-10-30T18:22:17Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-30T18:22:17Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-30T18:22:17Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-30T18:22:17Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-133-47.ec2.internal\": Patch \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/api/v1/nodes/ip-10-0-133-47.ec2.internal/status?timeout=10s\": context deadline exceeded" Oct 30 18:22:28.655611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:28.655577 5624 generic.go:356] "Generic (PLEG): container finished" podID="8626ce00-6a12-48e5-a761-c20623fd714a" containerID="6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25" exitCode=0 Oct 30 18:22:28.655976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:28.655659 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" event={"ID":"8626ce00-6a12-48e5-a761-c20623fd714a","Type":"ContainerDied","Data":"6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25"} Oct 30 18:22:28.656280 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:28.656262 5624 scope.go:122] "RemoveContainer" containerID="6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25" Oct 30 18:22:28.658606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:28.658576 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:28.659410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:28.659381 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:28.659485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:28.659441 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:29.399250 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.399208 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:22:29.563993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.563956 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564036 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2x585\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564080 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564124 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564162 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8kj45\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564195 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564232 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564272 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564310 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564349 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564423 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564464 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.564495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564488 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564506 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564539 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fb0abbf7-dockercfg-z4rks\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564555 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564579 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"default-dockercfg-5qwn9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564588 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564617 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fde2c5-openshift-default-dockercfg-ffxp6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564427 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564667 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564681 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ae3373-openshift-default-dockercfg-kcj2c\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564705 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564739 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gpm9w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564756 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"distinct-w1-fb2fa0-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564778 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564793 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564813 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564815 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564884 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564985 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e5d168-openshift-default-dockercfg-hlbp7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564987 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565035 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565040 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"premium-w4-4ef6ac-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565091 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b5sg4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565116 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565132 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"default-dockercfg-zff9h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565177 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565190 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565214 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565230 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565253 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565272 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565310 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565357 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565381 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565395 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b2b596\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565418 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565435 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-273105-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565466 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565475 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fdf4da-openshift-default-dockercfg-gnh4d\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564618 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jbvcp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565528 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"default-dockercfg-j6p62\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565534 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b2b596-dockercfg-6vvdt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565572 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"distinct-2-w0-6ce5eb-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565586 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"default-dockercfg-d99qs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565609 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.564707 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565625 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"premium-w2-fde2c5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565655 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565666 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565696 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"distinct-w5-fdf4da-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565730 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f9bu29dqg7ac4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565738 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8b6cf1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565769 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565778 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565815 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"default-dockercfg-vlhvr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565809 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565854 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565858 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565882 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fb0abbf7-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565899 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-fde2c5-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565912 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565993 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"simulator-w6-ef13cc-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565999 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566029 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h6qwn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566032 5624 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566060 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566085 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566098 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566118 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566138 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566169 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566201 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566232 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566240 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fb0abbf7\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566256 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566271 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566288 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566304 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.566267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566324 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566357 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566390 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.565696 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566424 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-273105-dockercfg-4zjdw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566461 5624 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566480 5624 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566170 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ef6ac-openshift-default-dockercfg-vxmbz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566500 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6ce5eb-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566401 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566561 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566570 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566204 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566430 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566607 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566622 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566643 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dcw6s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566717 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566721 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-t6khc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566754 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jf5s7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566768 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566790 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"premium-w0-6ce5eb-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566802 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566819 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mdm7d\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566839 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566842 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-273105\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566875 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:29.566764 5624 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/events/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft.18e361a12cbfe9b4\": http2: client connection lost" event="&Event{ObjectMeta:{e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft.18e361a12cbfe9b4 openshift-ingress 53377 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-ingress,Name:e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft,UID:47e369dc-6ca5-45f2-9c24-00d8a7421900,APIVersion:v1,ResourceVersion:53316,FieldPath:spec.containers{istio-proxy},},Reason:Pulled,Message:Container image \"registry.redhat.io/openshift-service-mesh/istio-proxyv2-rhel9@sha256:1fc05d17a76d31503f68c5bde3057750a2c575981535756112c06589f88c8c55\" already present on machine and can be accessed by the pod,Source:EventSource{Component:kubelet,Host:ip-10-0-133-47.ec2.internal,},FirstTimestamp:2026-10-30 18:20:21 +0000 UTC,LastTimestamp:2026-10-30 18:21:53.042654851 +0000 UTC m=+1207.431182968,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-47.ec2.internal,}" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566888 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hhk6k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566917 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxwr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566924 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2fa0-openshift-default-dockercfg-vw5sp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566926 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566984 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566980 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"default-dockercfg-45pmf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567014 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9ptn6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.566988 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567048 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b2b596-openshift-default-dockercfg-x4hhc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567062 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4865w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567070 5624 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567014 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567085 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567098 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567116 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567123 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ce5eb-openshift-default-dockercfg-td7nf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567131 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567154 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567158 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567167 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567189 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567195 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567202 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567223 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567231 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:29.567250 5624 controller.go:251] "Failed to update lease" err="Put \"https://a12840c7323dd4f5e8ca7b6f319d95c3-aafe87cc6257ca1d.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-133-47.ec2.internal?timeout=10s\": http2: client connection lost" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567266 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567274 5624 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567260 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567297 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-xm7rs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567310 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q92vx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567341 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4ef6ac-dockercfg-rhmzx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567349 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567375 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567388 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567410 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8b6cf1-dockercfg-mmwdg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567426 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwrgm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567446 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jwv4k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567468 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567481 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567504 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567515 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-726w6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567541 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567541 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567564 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18mfosc96e6jj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567587 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4ef6ac\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567567 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6ce5eb\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567627 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567650 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567658 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567682 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567707 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-8g42t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567729 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jv9hg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567746 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567767 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567793 5624 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567800 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567814 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567837 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567866 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567879 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567890 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b6cf1-openshift-default-dockercfg-l4bh4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567753 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567957 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567963 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6ce5eb-dockercfg-hs7vt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.567997 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568006 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568030 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568037 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gzppf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568065 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568076 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568099 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef13cc-openshift-default-dockercfg-5k66q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568117 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-273105-openshift-default-dockercfg-p57lq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568139 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568166 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568174 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568202 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568207 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568238 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568270 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568274 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568239 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568304 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.568982 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.569035 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.569071 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.569105 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.639849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.569170 5624 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 18:22:29.670596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.670522 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" event={"ID":"8626ce00-6a12-48e5-a761-c20623fd714a","Type":"ContainerStarted","Data":"9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e"} Oct 30 18:22:29.979229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.979125 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:22:30.023016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:29.979634 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:22:30.023016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.005191 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 18:22:30.145628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.145543 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:30.268420 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.268318 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:30.418963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.388258 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-273105-dockercfg-4zjdw\"" Oct 30 18:22:30.418963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.401237 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dcw6s\"" Oct 30 18:22:30.418963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.401355 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.418963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.418360 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" Oct 30 18:22:30.418963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.418469 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.418963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.418555 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-t6khc\"" Oct 30 18:22:30.434580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.434544 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.434759 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.434666 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 18:22:30.436839 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.436814 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.449314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.449290 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-fde2c5-serving-cert\"" Oct 30 18:22:30.449464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.449434 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 18:22:30.449536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.449482 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-xm7rs\"" Oct 30 18:22:30.449536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.449535 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:22:30.451997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.451971 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:22:30.479399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.479292 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.479399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.479364 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.479617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.479420 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2fa0-openshift-default-dockercfg-vw5sp\"" Oct 30 18:22:30.479617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.479483 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8g42t\"" Oct 30 18:22:30.479735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.479652 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:22:30.482306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.482282 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:22:30.508728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.502250 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.508728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.502414 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"distinct-w1-fb2fa0-kserve-self-signed-certs\"" Oct 30 18:22:30.508728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.502510 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" Oct 30 18:22:30.508728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.502601 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8kj45\"" Oct 30 18:22:30.508728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.502691 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 18:22:30.536825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.536750 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" Oct 30 18:22:30.537055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.537037 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b2b596-openshift-default-dockercfg-x4hhc\"" Oct 30 18:22:30.537162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.537145 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 18:22:30.537220 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.537212 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:22:30.537355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.537346 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:22:30.537429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.537403 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:22:30.537484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.537464 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" Oct 30 18:22:30.537543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.537510 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:22:30.537592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.537559 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:22:30.549312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.549288 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.549614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.549598 5624 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:22:30.550200 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.550184 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.550269 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.550242 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fdf4da-openshift-default-dockercfg-gnh4d\"" Oct 30 18:22:30.550386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.550370 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ce5eb-openshift-default-dockercfg-td7nf\"" Oct 30 18:22:30.563634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.563606 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" Oct 30 18:22:30.564705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.564683 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.568769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.568751 5624 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:22:30.570816 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.569331 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:22:30.576795 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.576750 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"premium-w2-fde2c5-kserve-self-signed-certs\"" Oct 30 18:22:30.578813 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.578795 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.579378 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.578885 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fb0abbf7-dockercfg-z4rks\"" Oct 30 18:22:30.637027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.592233 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 18:22:30.637027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.630171 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:22:30.637027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.631926 5624 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:22:30.637027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.631974 5624 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:22:30.637027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.631989 5624 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:22:30.637027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.632017 5624 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-47.ec2.internal" event="NodeReady" Oct 30 18:22:30.648322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.648294 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 18:22:30.648601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.648582 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:22:30.648887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.648867 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4865w\"" Oct 30 18:22:30.649104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.649089 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.649345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.649329 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:22:30.652133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.652115 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-273105-serving-cert\"" Oct 30 18:22:30.653120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.653104 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h6qwn\"" Oct 30 18:22:30.659436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.659414 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" Oct 30 18:22:30.662399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.662378 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.677834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.677810 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:22:30.678290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.678271 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.678385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.678372 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f9bu29dqg7ac4\"" Oct 30 18:22:30.678489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.678475 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 18:22:30.706418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.706393 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-726w6\"" Oct 30 18:22:30.723628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.706721 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e5d168-openshift-default-dockercfg-hlbp7\"" Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.720646 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.720702 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.721233 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.721705 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.721781 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.721828 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.721914 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.722006 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gzppf\"" Oct 30 18:22:30.734396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.720047 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.739796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.739773 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:22:30.741578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.740050 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-273105\"" Oct 30 18:22:30.741844 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.740160 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 18:22:30.741844 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.740237 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.742015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.740591 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.742015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.740703 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fb0abbf7-serving-cert\"" Oct 30 18:22:30.742015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.741703 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 18:22:30.755016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.754977 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-273105-openshift-default-dockercfg-p57lq\"" Oct 30 18:22:30.755494 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.755348 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.755494 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.755425 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.755860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.755763 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.755860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.755834 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.757571 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.757423 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.757571 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.757537 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fb0abbf7\"" Oct 30 18:22:30.761487 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.761387 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b2b596-dockercfg-6vvdt\"" Oct 30 18:22:30.761739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.761704 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.767163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.767148 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.767383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.767147 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:22:30.769196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.769179 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.771576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.771558 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.771892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.771877 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-18mfosc96e6jj\"" Oct 30 18:22:30.780902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.780880 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.783384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.783369 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6ce5eb-serving-cert\"" Oct 30 18:22:30.815266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.815188 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.815410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.815362 5624 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:22:30.815486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.815427 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 18:22:30.815548 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.815486 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:22:30.830096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.830078 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:22:30.830617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.830603 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:22:30.830972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.830952 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jv9hg\"" Oct 30 18:22:30.831170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.831157 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"default-dockercfg-zff9h\"" Oct 30 18:22:30.831429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.831413 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:22:30.831626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.831492 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b2b596\"" Oct 30 18:22:30.831739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.831561 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxwr\"" Oct 30 18:22:30.831812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.831579 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:22:30.831877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.831662 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jwv4k\"" Oct 30 18:22:30.836506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.836488 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6ce5eb\"" Oct 30 18:22:30.838803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.838786 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 18:22:30.839812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.839778 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.840115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.840099 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:22:30.844171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.844149 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.846513 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.846496 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 18:22:30.846749 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.846731 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6ce5eb-dockercfg-hs7vt\"" Oct 30 18:22:30.863479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.861509 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mdm7d\"" Oct 30 18:22:30.863479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.861627 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:22:30.863479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.861736 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"premium-w4-4ef6ac-kserve-self-signed-certs\"" Oct 30 18:22:30.869276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.869251 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 18:22:30.869591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.869575 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:22:30.896211 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.889350 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:22:30.925557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.907271 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.925557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.907355 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.925557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.907539 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8b6cf1-dockercfg-mmwdg\"" Oct 30 18:22:30.925557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.907622 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" Oct 30 18:22:30.925557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.907727 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"distinct-2-w0-6ce5eb-kserve-self-signed-certs\"" Oct 30 18:22:30.926336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.926308 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:22:30.926600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.926580 5624 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:22:30.926706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.926690 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"openshift-service-ca.crt\"" Oct 30 18:22:30.926849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.926830 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hhk6k\"" Oct 30 18:22:30.926923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.926912 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:22:30.927077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.927060 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 18:22:30.930956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.928358 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:22:30.930956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.928500 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"default-dockercfg-5qwn9\"" Oct 30 18:22:30.930956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.928596 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"premium-w0-6ce5eb-kserve-self-signed-certs\"" Oct 30 18:22:30.930956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.928637 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" Oct 30 18:22:30.931246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.931226 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fde2c5-openshift-default-dockercfg-ffxp6\"" Oct 30 18:22:31.032008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.961102 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:22:31.032242 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:30.961197 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"openshift-service-ca.crt\"" Oct 30 18:22:31.032346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005246 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"simulator-w6-ef13cc-kserve-self-signed-certs\"" Oct 30 18:22:31.032426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005303 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:22:31.032505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005323 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:22:31.032644 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.032622 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 18:22:31.032755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005431 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.032840 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005393 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:22:31.032927 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005497 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gpm9w\"" Oct 30 18:22:31.033033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005559 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.033129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005606 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:22:31.033218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005661 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:22:31.033315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005666 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 18:22:31.033404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.025203 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:22:31.033493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.025617 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2x585\"" Oct 30 18:22:31.033493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.025713 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:22:31.033595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.025775 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.033595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.025824 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8b6cf1\"" Oct 30 18:22:31.033595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.025869 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:22:31.033595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.026037 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:22:31.033595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.029343 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:22:31.033595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.005513 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:22:31.038756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.038735 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.038987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.038909 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:22:31.039087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.039071 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.039300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.039284 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwrgm\"" Oct 30 18:22:31.039391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.039368 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" Oct 30 18:22:31.039677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.039665 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:22:31.048810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.048788 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 18:22:31.051037 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.051019 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:22:31.051659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.051644 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 18:22:31.053822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.053795 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4ef6ac\"" Oct 30 18:22:31.060148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.060130 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:22:31.066776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.060409 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"default-dockercfg-45pmf\"" Oct 30 18:22:31.071826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.071795 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"default-dockercfg-d99qs\"" Oct 30 18:22:31.072007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.071882 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4ef6ac-dockercfg-rhmzx\"" Oct 30 18:22:31.077288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.077090 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 18:22:31.077288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.077170 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" Oct 30 18:22:31.077288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.077211 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef13cc-openshift-default-dockercfg-5k66q\"" Oct 30 18:22:31.077288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.077272 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jbvcp\"" Oct 30 18:22:31.078314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.078277 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b5sg4\"" Oct 30 18:22:31.078421 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.078406 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" Oct 30 18:22:31.082896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.082872 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"default-dockercfg-j6p62\"" Oct 30 18:22:31.111958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.089642 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jf5s7\"" Oct 30 18:22:31.113394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.113367 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:22:31.114254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.114233 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9ptn6\"" Oct 30 18:22:31.114511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.114495 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.114723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.114709 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ae3373-openshift-default-dockercfg-kcj2c\"" Oct 30 18:22:31.123953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.120301 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:22:31.123953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.120384 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:22:31.123953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.120482 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b6cf1-openshift-default-dockercfg-l4bh4\"" Oct 30 18:22:31.123953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.120537 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:22:31.153721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.153689 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ef6ac-openshift-default-dockercfg-vxmbz\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.171460 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-fdf4da\"/\"distinct-w5-fdf4da-kserve-self-signed-certs\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.172174 5624 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.172202 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.172591 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.173108 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.173207 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.173477 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.173780 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.173874 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.174329 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.174629 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:22:31.175097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.174677 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q92vx\"" Oct 30 18:22:31.187956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.185091 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:22:31.187956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.185358 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:22:31.200983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.196355 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:22:31.262043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.262013 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"default-dockercfg-vlhvr\"" Oct 30 18:22:31.265727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.265693 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f37fa868-g5zcm"] Oct 30 18:22:31.265911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.265764 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr","openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7","openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr"] Oct 30 18:22:31.271854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.271825 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.276983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.276560 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.280982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.280952 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:22:31.281124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.281014 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p"] Oct 30 18:22:31.281124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.281058 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.281370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.281340 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" containerID="cri-o://786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f" gracePeriod=30 Oct 30 18:22:31.312999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.312194 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9af5623f-openshift-default-dockercfg-vxdf5\"" Oct 30 18:22:31.419781 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.419745 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.420364 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.420342 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.420534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.420520 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.420659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.420646 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.420791 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.420778 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.420982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.420966 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.421461 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421437 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sd22\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-kube-api-access-8sd22\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.421578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421559 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.421635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421601 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.421688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421634 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea5d15c-4957-49c0-9739-f5383e23f424-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.421688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421660 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjfj\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-kube-api-access-krjfj\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.421792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421694 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.421792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421728 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea5d15c-4957-49c0-9739-f5383e23f424-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.421792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421775 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.421819 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423176 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423204 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8tg\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-kube-api-access-gv8tg\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423236 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423264 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423319 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423368 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423394 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423440 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423468 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423511 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423548 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.423585 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.525425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525385 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.525623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525458 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.525623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525515 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.525623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525598 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.525804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525635 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sd22\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-kube-api-access-8sd22\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.525804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525702 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.525804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525736 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.525804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525767 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea5d15c-4957-49c0-9739-f5383e23f424-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.525804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525797 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjfj\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-kube-api-access-krjfj\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.526110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525822 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.526110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525831 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.526110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525873 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea5d15c-4957-49c0-9739-f5383e23f424-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.526110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.525922 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.526110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526012 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.526110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526051 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.526110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526079 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8tg\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-kube-api-access-gv8tg\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.526473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526117 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.526473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526147 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.526473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526153 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.526473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526444 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.526684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526479 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.526842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526818 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.526924 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526898 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.527010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526949 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.527010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.526996 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.527116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527031 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.527116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527042 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.527116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527054 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.527116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527108 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527151 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527415 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527501 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527516 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea5d15c-4957-49c0-9739-f5383e23f424-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527661 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527154 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527761 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527762 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527815 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.527826 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.528111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.528091 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.528702 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.528265 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.529730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.529702 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.531836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.531783 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.531836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.531792 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.531836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.531804 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.532241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.532204 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.532923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.532686 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea5d15c-4957-49c0-9739-f5383e23f424-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.534006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.533961 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.558415 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.558372 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjfj\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-kube-api-access-krjfj\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.558870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.558838 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.559296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.559270 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sd22\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-kube-api-access-8sd22\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.559354 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.559320 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8tg\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-kube-api-access-gv8tg\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.559531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.559516 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.560664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.560642 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.621141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.621056 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:31.638832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.638808 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:31.645704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.645681 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:31.846792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.846752 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab5896a-19cb-4175-a41a-3f1cc183990c" path="/var/lib/kubelet/pods/1ab5896a-19cb-4175-a41a-3f1cc183990c/volumes" Oct 30 18:22:31.979175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.979062 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:22:31.979175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:31.979121 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 18:22:32.046382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.046342 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr"] Oct 30 18:22:32.138456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.124030 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:32.138456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.135995 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr"] Oct 30 18:22:32.146818 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.146779 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.147058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.147035 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.147196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.147178 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.158715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.152619 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7"] Oct 30 18:22:32.158715 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:22:32.152968 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b4e3bb_6754_4431_bd4a_c2b1512457ba.slice/crio-934f1bd7da4ee3ab815c9f0e4428052b68527382488969f2c03da913ca193d02 WatchSource:0}: Error finding container 934f1bd7da4ee3ab815c9f0e4428052b68527382488969f2c03da913ca193d02: Status 404 returned error can't find the container with id 934f1bd7da4ee3ab815c9f0e4428052b68527382488969f2c03da913ca193d02 Oct 30 18:22:32.163796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.163749 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.164086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.163836 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.164086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.163893 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.164376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.164338 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.164521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.164456 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.164521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.164510 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:32.337410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.337310 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:32.766595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.765372 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" event={"ID":"fea5d15c-4957-49c0-9739-f5383e23f424","Type":"ContainerStarted","Data":"cc434f884a702dd2d77b7578742773d3dd3d5fe5c964b602f959a2cff97c725a"} Oct 30 18:22:32.766595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.765435 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" event={"ID":"fea5d15c-4957-49c0-9739-f5383e23f424","Type":"ContainerStarted","Data":"f05fb32340d97894e911163000e528fd9b22972e9008c0c58e6c2e2ec1f681df"} Oct 30 18:22:32.769480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.768650 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" event={"ID":"2525bc9c-24dd-4ee0-afe9-2d79c0266d41","Type":"ContainerStarted","Data":"a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9"} Oct 30 18:22:32.769480 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.768697 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" event={"ID":"2525bc9c-24dd-4ee0-afe9-2d79c0266d41","Type":"ContainerStarted","Data":"8f4b507247e93eda430067214e6168866628bd2d3fcf79ee5e3eaf39c91f6bf9"} Oct 30 18:22:32.772780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.772752 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" event={"ID":"f5b4e3bb-6754-4431-bd4a-c2b1512457ba","Type":"ContainerStarted","Data":"a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6"} Oct 30 18:22:32.772879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.772803 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" event={"ID":"f5b4e3bb-6754-4431-bd4a-c2b1512457ba","Type":"ContainerStarted","Data":"934f1bd7da4ee3ab815c9f0e4428052b68527382488969f2c03da913ca193d02"} Oct 30 18:22:32.849116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.849065 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" podStartSLOduration=45.849050005 podStartE2EDuration="45.849050005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:32.83116261 +0000 UTC m=+1247.219690730" watchObservedRunningTime="2026-10-30 18:22:32.849050005 +0000 UTC m=+1247.237578122" Oct 30 18:22:32.877839 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.877775 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podStartSLOduration=12.877762663 podStartE2EDuration="12.877762663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:32.875725663 +0000 UTC m=+1247.264253781" watchObservedRunningTime="2026-10-30 18:22:32.877762663 +0000 UTC m=+1247.266290779" Oct 30 18:22:32.961967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:32.957962 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" podStartSLOduration=6.957922432 podStartE2EDuration="6.957922432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:32.956821029 +0000 UTC m=+1247.345349146" watchObservedRunningTime="2026-10-30 18:22:32.957922432 +0000 UTC m=+1247.346450551" Oct 30 18:22:33.019012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.013482 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:22:33.019012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.013546 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 18:22:33.622558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.622368 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:33.629369 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.629337 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:33.639601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.639581 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:33.644539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.644519 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:33.646733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.646719 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:33.647723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.647704 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 18:22:33.647785 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.647746 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 18:22:33.800304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.800274 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:33.800824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.800809 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:33.800979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.800967 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:22:33.801134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.801123 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:22:33.979530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.979447 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:22:33.979530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:33.979511 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 18:22:34.245733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:34.245651 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:34.651231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:34.651203 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:34.823544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:34.823510 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:34.824710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:34.824686 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:22:35.013857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.005800 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:22:35.013857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.005856 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 18:22:35.390025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.388398 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:35.490743 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.490692 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:35.665743 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.665663 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv"] Oct 30 18:22:35.675896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.675866 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.728892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.716491 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv"] Oct 30 18:22:35.820403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.820372 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.820590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.820422 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.820590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.820541 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.820687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.820597 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.820767 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.820742 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46237646-21b2-4830-9635-82d792e261c5-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.820852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.820835 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.821466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.820918 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8bq\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-kube-api-access-mm8bq\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.821466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.820993 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46237646-21b2-4830-9635-82d792e261c5-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.821466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.821050 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.923274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.923177 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.923758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.923288 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46237646-21b2-4830-9635-82d792e261c5-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.923758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.923389 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.924826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.924796 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.926330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.925879 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8bq\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-kube-api-access-mm8bq\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.926330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.926001 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46237646-21b2-4830-9635-82d792e261c5-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.926330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.926117 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.926564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.926335 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.926564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.926403 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.926564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.926539 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.929481 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.927048 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.931753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.931712 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46237646-21b2-4830-9635-82d792e261c5-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.933858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.933821 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.934300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.934282 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.934889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.934868 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.947453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.947429 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46237646-21b2-4830-9635-82d792e261c5-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.961288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.961257 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8bq\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-kube-api-access-mm8bq\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:35.963026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.963002 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:36.001075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.985419 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:22:36.001075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.987006 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:22:36.001075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:35.988289 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:22:36.037753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.037714 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:36.430102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.429342 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv"] Oct 30 18:22:36.443095 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:22:36.443054 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46237646_21b2_4830_9635_82d792e261c5.slice/crio-72e4ab473b80e22ac571663a158d906fa1d2a169b160abe10581cdc4c097511f WatchSource:0}: Error finding container 72e4ab473b80e22ac571663a158d906fa1d2a169b160abe10581cdc4c097511f: Status 404 returned error can't find the container with id 72e4ab473b80e22ac571663a158d906fa1d2a169b160abe10581cdc4c097511f Oct 30 18:22:36.446642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.446607 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:36.446756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.446677 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:36.446756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.446705 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:36.589513 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.589366 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:36.601317 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.601293 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:22:36.646360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646330 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-certs\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.646537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646383 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-envoy\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.646537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646422 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4740229c-ee94-4644-8678-66ab753340dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4740229c-ee94-4644-8678-66ab753340dd-istiod-ca-cert\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.646537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646509 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-kube-api-access-8xfkk\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-kube-api-access-8xfkk\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.646726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646548 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-data\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.646726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646624 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-socket\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.646726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646642 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-certs" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:36.646894 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646785 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-istio-token\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-istio-token\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.646989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646895 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4740229c-ee94-4644-8678-66ab753340dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4740229c-ee94-4644-8678-66ab753340dd-istio-podinfo\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.647048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.646984 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-data" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:36.647048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.647024 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4740229c-ee94-4644-8678-66ab753340dd-istiod-ca-cert" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:36.647153 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.647116 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-credential-socket\") pod \"4740229c-ee94-4644-8678-66ab753340dd\" (UID: \"4740229c-ee94-4644-8678-66ab753340dd\") " Oct 30 18:22:36.647567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.647540 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-credential-socket" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:36.647758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.647602 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-socket" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:36.648223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.648201 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.648223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.648226 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.648393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.648235 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4740229c-ee94-4644-8678-66ab753340dd-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.648393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.648248 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.648393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.648261 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.650391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.650355 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4740229c-ee94-4644-8678-66ab753340dd-istio-podinfo" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:36.650664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.650642 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-envoy" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:36.650862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.650844 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-istio-token" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:36.651283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.651266 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-kube-api-access-8xfkk" pod "4740229c-ee94-4644-8678-66ab753340dd" (UID: "4740229c-ee94-4644-8678-66ab753340dd"). InnerVolumeSpecName "kube-api-access-8xfkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:36.755800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.755723 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4740229c-ee94-4644-8678-66ab753340dd-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.755800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.755761 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xfkk\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-kube-api-access-8xfkk\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.755800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.755771 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4740229c-ee94-4644-8678-66ab753340dd-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.755800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.755779 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4740229c-ee94-4644-8678-66ab753340dd-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:36.854004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.853898 5624 generic.go:356] "Generic (PLEG): container finished" podID="4740229c-ee94-4644-8678-66ab753340dd" containerID="786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f" exitCode=0 Oct 30 18:22:36.854174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.854073 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" event={"ID":"4740229c-ee94-4644-8678-66ab753340dd","Type":"ContainerDied","Data":"786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f"} Oct 30 18:22:36.854174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.854127 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" event={"ID":"4740229c-ee94-4644-8678-66ab753340dd","Type":"ContainerDied","Data":"9ea0b3f31acb38c7891e28f8eaf8259e2db56fa2dc30f9a40e02bed447085ecd"} Oct 30 18:22:36.854174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.854158 5624 scope.go:122] "RemoveContainer" containerID="786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f" Oct 30 18:22:36.854348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.854327 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p" Oct 30 18:22:36.865968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.865368 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" event={"ID":"46237646-21b2-4830-9635-82d792e261c5","Type":"ContainerStarted","Data":"57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b"} Oct 30 18:22:36.865968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.865424 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" event={"ID":"46237646-21b2-4830-9635-82d792e261c5","Type":"ContainerStarted","Data":"72e4ab473b80e22ac571663a158d906fa1d2a169b160abe10581cdc4c097511f"} Oct 30 18:22:36.875978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.875862 5624 scope.go:122] "RemoveContainer" containerID="786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f" Oct 30 18:22:36.877036 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:36.877006 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f\": container with ID starting with 786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f not found: ID does not exist" containerID="786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f" Oct 30 18:22:36.877143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:36.877045 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f"} err="failed to get container status \"786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f\": rpc error: code = NotFound desc = could not find container \"786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f\": container with ID starting with 786daf8d623b000ec19bfbac6544e1d0a5920adbbf334e6dba96d96a64670c6f not found: ID does not exist" Oct 30 18:22:37.044185 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.044150 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:37.078829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.063149 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Oct 30 18:22:37.079231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.079194 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" podUID="46237646-21b2-4830-9635-82d792e261c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Oct 30 18:22:37.082701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.066121 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:37.333422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.328385 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:37.453184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.453129 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:37.533364 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.533327 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd"] Oct 30 18:22:37.534282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.534261 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" Oct 30 18:22:37.534282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.534282 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" Oct 30 18:22:37.534469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.534458 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4740229c-ee94-4644-8678-66ab753340dd" containerName="istio-proxy" Oct 30 18:22:37.539815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.539794 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:37.566264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.566245 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9af5623f-dockercfg-tm2bt\"" Oct 30 18:22:37.566453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.566436 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9af5623f\"" Oct 30 18:22:37.568190 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.568169 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6"] Oct 30 18:22:37.572913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.572894 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:37.636784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.636738 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6"] Oct 30 18:22:37.642605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.642574 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9m4b\" (UniqueName: \"kubernetes.io/projected/0ffdb840-fd18-4ad2-8635-2234ab2069b0-kube-api-access-j9m4b\") pod \"payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd\" (UID: \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:37.642770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.642617 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km2xb\" (UniqueName: \"kubernetes.io/projected/13b40734-590f-4568-b812-c77303a52d56-kube-api-access-km2xb\") pod \"payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6\" (UID: \"13b40734-590f-4568-b812-c77303a52d56\") " pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:37.647002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.642995 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b40734-590f-4568-b812-c77303a52d56-plugins-config-volume\") pod \"payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6\" (UID: \"13b40734-590f-4568-b812-c77303a52d56\") " pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:37.647002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.643042 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffdb840-fd18-4ad2-8635-2234ab2069b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd\" (UID: \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:37.647002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.646609 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd"] Oct 30 18:22:37.745565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.745531 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b40734-590f-4568-b812-c77303a52d56-plugins-config-volume\") pod \"payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6\" (UID: \"13b40734-590f-4568-b812-c77303a52d56\") " pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:37.745754 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.745585 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffdb840-fd18-4ad2-8635-2234ab2069b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd\" (UID: \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:37.745829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.745802 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9m4b\" (UniqueName: \"kubernetes.io/projected/0ffdb840-fd18-4ad2-8635-2234ab2069b0-kube-api-access-j9m4b\") pod \"payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd\" (UID: \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:37.745890 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.745846 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km2xb\" (UniqueName: \"kubernetes.io/projected/13b40734-590f-4568-b812-c77303a52d56-kube-api-access-km2xb\") pod \"payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6\" (UID: \"13b40734-590f-4568-b812-c77303a52d56\") " pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:37.746247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.746218 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b40734-590f-4568-b812-c77303a52d56-plugins-config-volume\") pod \"payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6\" (UID: \"13b40734-590f-4568-b812-c77303a52d56\") " pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:37.746721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.746701 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffdb840-fd18-4ad2-8635-2234ab2069b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd\" (UID: \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:37.768105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.768076 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km2xb\" (UniqueName: \"kubernetes.io/projected/13b40734-590f-4568-b812-c77303a52d56-kube-api-access-km2xb\") pod \"payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6\" (UID: \"13b40734-590f-4568-b812-c77303a52d56\") " pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:37.768345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.768317 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9m4b\" (UniqueName: \"kubernetes.io/projected/0ffdb840-fd18-4ad2-8635-2234ab2069b0-kube-api-access-j9m4b\") pod \"payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd\" (UID: \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:37.917006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.909652 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:37.917006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:37.912298 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:38.073331 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:38.073297 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:38.129676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:38.129644 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:38.136763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:38.136730 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:22:38.725998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:38.725493 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd"] Oct 30 18:22:38.739247 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:22:38.739152 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ffdb840_fd18_4ad2_8635_2234ab2069b0.slice/crio-55f1067b89d51a8a529d8d3bba1e770f5ee62bea09ce6c06588fc1acf17e389d WatchSource:0}: Error finding container 55f1067b89d51a8a529d8d3bba1e770f5ee62bea09ce6c06588fc1acf17e389d: Status 404 returned error can't find the container with id 55f1067b89d51a8a529d8d3bba1e770f5ee62bea09ce6c06588fc1acf17e389d Oct 30 18:22:38.937531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:38.937501 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6"] Oct 30 18:22:39.026776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:39.026736 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:39.127706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:39.127669 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:39.142367 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:39.142332 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" event={"ID":"0ffdb840-fd18-4ad2-8635-2234ab2069b0","Type":"ContainerStarted","Data":"783b130c6019ae2d1c7ce699417eaad2ac8529751b4a3304210ee5ec3330a9f7"} Oct 30 18:22:39.142542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:39.142400 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:39.142542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:39.142421 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" event={"ID":"0ffdb840-fd18-4ad2-8635-2234ab2069b0","Type":"ContainerStarted","Data":"55f1067b89d51a8a529d8d3bba1e770f5ee62bea09ce6c06588fc1acf17e389d"} Oct 30 18:22:39.144324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:39.144296 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" event={"ID":"13b40734-590f-4568-b812-c77303a52d56","Type":"ContainerStarted","Data":"4551d941857a4fd7a01c32642a98b30d40486b3176292df7258bf8f6f2091978"} Oct 30 18:22:39.144550 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:39.144517 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" event={"ID":"13b40734-590f-4568-b812-c77303a52d56","Type":"ContainerStarted","Data":"bcc4bf408b7c6b61809cc2751c0cd7eca5b54aab8e5c2d5e4fff1b10a541bc9d"} Oct 30 18:22:39.144672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:39.144560 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:41.183371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:41.183330 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:41.407958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:41.402440 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2"] Oct 30 18:22:41.407958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:41.402861 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" containerID="cri-o://9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a" gracePeriod=30 Oct 30 18:22:41.603408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:41.603314 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:41.928234 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:41.928144 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" podStartSLOduration=6.9281240870000005 podStartE2EDuration="6.928124087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:41.918587061 +0000 UTC m=+1256.307115178" watchObservedRunningTime="2026-10-30 18:22:41.928124087 +0000 UTC m=+1256.316652205" Oct 30 18:22:41.970709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:41.970661 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" podStartSLOduration=4.9706473110000005 podStartE2EDuration="4.970647311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:41.969194547 +0000 UTC m=+1256.357722665" watchObservedRunningTime="2026-10-30 18:22:41.970647311 +0000 UTC m=+1256.359175427" Oct 30 18:22:42.021098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:42.021036 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" podStartSLOduration=5.021019753 podStartE2EDuration="5.021019753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:42.0157093 +0000 UTC m=+1256.404237434" watchObservedRunningTime="2026-10-30 18:22:42.021019753 +0000 UTC m=+1256.409547871" Oct 30 18:22:42.204607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:42.204516 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p"] Oct 30 18:22:42.213770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:42.213734 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-xqs7p"] Oct 30 18:22:42.988430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:42.988393 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:43.165506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:43.165468 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:22:43.165677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:43.165538 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:22:43.378984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:43.378546 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:43.827183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:43.827114 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4740229c-ee94-4644-8678-66ab753340dd" path="/var/lib/kubelet/pods/4740229c-ee94-4644-8678-66ab753340dd/volumes" Oct 30 18:22:46.764242 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.764211 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:22:46.868058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868026 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-kube-api-access-59z8w\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-kube-api-access-59z8w\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.868231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868185 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-token\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-token\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.868231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868209 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-podinfo\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.868321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868286 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-envoy\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.868321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868315 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-socket\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.868482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868444 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-credential-socket\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.868539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868520 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5293e2-5b63-430f-aa36-9733a6a35dd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5293e2-5b63-430f-aa36-9733a6a35dd3-istiod-ca-cert\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.869087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868661 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-certs\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.869087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868693 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-data\") pod \"df5293e2-5b63-430f-aa36-9733a6a35dd3\" (UID: \"df5293e2-5b63-430f-aa36-9733a6a35dd3\") " Oct 30 18:22:46.869087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868783 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-credential-socket" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.869087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.868874 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-socket" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.869087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.869045 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-certs" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.869087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.869057 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5293e2-5b63-430f-aa36-9733a6a35dd3-istiod-ca-cert" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:46.870538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.869197 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-data" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.870538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.869893 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.870538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.869915 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.870538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.869930 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5293e2-5b63-430f-aa36-9733a6a35dd3-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.870538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.869960 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.870538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.869974 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.872337 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.871440 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-token" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:46.872337 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.871893 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-envoy" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.872337 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.871966 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-kube-api-access-59z8w" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "kube-api-access-59z8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:46.872671 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.872647 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-podinfo" pod "df5293e2-5b63-430f-aa36-9733a6a35dd3" (UID: "df5293e2-5b63-430f-aa36-9733a6a35dd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:46.974265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.974180 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59z8w\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-kube-api-access-59z8w\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.974265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.974213 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.974265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.974226 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.974265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:46.974237 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5293e2-5b63-430f-aa36-9733a6a35dd3-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:22:47.316678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.316600 5624 generic.go:356] "Generic (PLEG): container finished" podID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerID="9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a" exitCode=0 Oct 30 18:22:47.316678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.316653 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" Oct 30 18:22:47.317373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.316678 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" event={"ID":"df5293e2-5b63-430f-aa36-9733a6a35dd3","Type":"ContainerDied","Data":"9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a"} Oct 30 18:22:47.317373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.316743 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2" event={"ID":"df5293e2-5b63-430f-aa36-9733a6a35dd3","Type":"ContainerDied","Data":"e325dd9539631d835e7b2cc42247c7620e517eba08667395f7f330e1e833f59d"} Oct 30 18:22:47.317373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.316774 5624 scope.go:122] "RemoveContainer" containerID="9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a" Oct 30 18:22:47.337540 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.337519 5624 scope.go:122] "RemoveContainer" containerID="0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52" Oct 30 18:22:47.351761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.351736 5624 scope.go:122] "RemoveContainer" containerID="9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a" Oct 30 18:22:47.352163 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:47.352139 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a\": container with ID starting with 9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a not found: ID does not exist" containerID="9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a" Oct 30 18:22:47.352261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.352174 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a"} err="failed to get container status \"9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a\": rpc error: code = NotFound desc = could not find container \"9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a\": container with ID starting with 9fce34c9cee085208635b9367cfd7c0479cf04800adeebf706573030003d216a not found: ID does not exist" Oct 30 18:22:47.352261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.352200 5624 scope.go:122] "RemoveContainer" containerID="0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52" Oct 30 18:22:47.352609 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:47.352574 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52\": container with ID starting with 0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52 not found: ID does not exist" containerID="0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52" Oct 30 18:22:47.352720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.352613 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52"} err="failed to get container status \"0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52\": rpc error: code = NotFound desc = could not find container \"0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52\": container with ID starting with 0d98d18638d70639ec69bfec07903addb37f58c8221b5cccea56bcfd6a302d52 not found: ID does not exist" Oct 30 18:22:47.392398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.392365 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2"] Oct 30 18:22:47.403996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.403959 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-wxkk2"] Oct 30 18:22:47.798730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:47.798687 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" path="/var/lib/kubelet/pods/df5293e2-5b63-430f-aa36-9733a6a35dd3/volumes" Oct 30 18:22:49.922541 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:22:49.922505 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:22:50.217922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.217841 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:22:50.218282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.218262 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:22:50.695700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.695661 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj"] Oct 30 18:22:50.696897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.696874 5624 cpu_manager.go:414] "containerMap: removing container" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" Oct 30 18:22:50.696897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.696909 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" Oct 30 18:22:50.697694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.697021 5624 cpu_manager.go:414] "containerMap: removing container" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" Oct 30 18:22:50.697694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.697030 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" Oct 30 18:22:50.697694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.697316 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" Oct 30 18:22:50.697694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.697343 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5293e2-5b63-430f-aa36-9733a6a35dd3" containerName="istio-proxy" Oct 30 18:22:50.703911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.703880 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.734544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.734514 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj"] Oct 30 18:22:50.843219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843184 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.843219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843221 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.843412 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843386 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.843452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843445 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.843530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843513 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.843655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843632 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.843749 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843728 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffkpn\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-kube-api-access-ffkpn\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.843804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843765 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.843912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.843896 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.945335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945300 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.945335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945340 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.945827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945418 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.945827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945513 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.945827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945549 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffkpn\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-kube-api-access-ffkpn\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.945827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945567 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.945827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945674 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.945827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945808 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.946187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945825 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.946187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.945843 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.946187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.946051 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.946187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.946122 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.946187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.946160 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.947034 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.946216 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.948006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.947983 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.948304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.948281 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.965842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.965817 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:50.966160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:50.966140 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffkpn\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-kube-api-access-ffkpn\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:51.045820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:51.045793 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:51.378809 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:22:51.378773 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17022b7e_05e9_4f50_a89b_ac0dc7e3057b.slice/crio-b81c76a7e3ae7433199ac5b763c610f58cd5f2a1de2d079cfb2a6d97a1d40b66 WatchSource:0}: Error finding container b81c76a7e3ae7433199ac5b763c610f58cd5f2a1de2d079cfb2a6d97a1d40b66: Status 404 returned error can't find the container with id b81c76a7e3ae7433199ac5b763c610f58cd5f2a1de2d079cfb2a6d97a1d40b66 Oct 30 18:22:51.382190 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:51.382149 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:51.382318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:51.382221 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:51.382318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:51.382252 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:22:51.383274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:51.383249 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj"] Oct 30 18:22:52.377347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:52.377307 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" event={"ID":"17022b7e-05e9-4f50-a89b-ac0dc7e3057b","Type":"ContainerStarted","Data":"44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e"} Oct 30 18:22:52.377759 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:52.377367 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" event={"ID":"17022b7e-05e9-4f50-a89b-ac0dc7e3057b","Type":"ContainerStarted","Data":"b81c76a7e3ae7433199ac5b763c610f58cd5f2a1de2d079cfb2a6d97a1d40b66"} Oct 30 18:22:52.437477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:52.437415 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" podStartSLOduration=2.437398954 podStartE2EDuration="2.437398954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:52.436671584 +0000 UTC m=+1266.825199701" watchObservedRunningTime="2026-10-30 18:22:52.437398954 +0000 UTC m=+1266.825927073" Oct 30 18:22:53.046050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:53.045978 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:53.047485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:53.047454 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:22:53.047641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:53.047513 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 18:22:54.032660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:54.032625 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v"] Oct 30 18:22:54.045609 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:54.045582 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dc722a-cbj4v"] Oct 30 18:22:54.046374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:54.046349 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:22:54.047003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:54.046396 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 18:22:55.046140 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:55.046100 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:22:55.046622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:55.046163 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 18:22:55.730951 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:55.728475 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:55.819424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:55.819388 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abc93f2-30f0-4fd9-bfd2-262775c596a3" path="/var/lib/kubelet/pods/4abc93f2-30f0-4fd9-bfd2-262775c596a3/volumes" Oct 30 18:22:56.046708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:56.046628 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:22:56.046708 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:56.046692 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 18:22:57.046850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:57.046818 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:22:57.047291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:57.046872 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 18:22:58.050870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:58.050839 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:58.051904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:58.051254 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:22:58.051904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:22:58.051759 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:23:04.258090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:04.258053 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:12.097053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.097016 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn"] Oct 30 18:23:12.103324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.103298 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.128396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.128362 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn"] Oct 30 18:23:12.177207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177169 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.177379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177265 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.177379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177345 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.177379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177372 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clv2l\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-kube-api-access-clv2l\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.178115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177428 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.178115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177595 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.178115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177877 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.178115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177909 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.178115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.177985 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.279347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279320 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.279521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279382 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.279592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279528 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.279592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279544 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.279592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279582 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.280351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279631 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.280351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279715 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.280351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279846 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.280351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279853 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.280351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279959 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.280351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279975 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.280351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.279991 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clv2l\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-kube-api-access-clv2l\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.280351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.280114 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.281076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.280523 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.285594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.285560 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.285910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.285888 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.299468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.299424 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.299591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.299561 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clv2l\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-kube-api-access-clv2l\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.439742 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.439709 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:12.779841 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:23:12.779807 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1773b5ea_e573_4ae6_a5d1_ea5e02a7c0b0.slice/crio-94bce3a27673e73f1c24ea81869cd96c758bd0fbd8de2940d2364fa2980efd2a WatchSource:0}: Error finding container 94bce3a27673e73f1c24ea81869cd96c758bd0fbd8de2940d2364fa2980efd2a: Status 404 returned error can't find the container with id 94bce3a27673e73f1c24ea81869cd96c758bd0fbd8de2940d2364fa2980efd2a Oct 30 18:23:12.782737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.782704 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:23:12.782817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.782771 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:23:12.782817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.782801 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:23:12.784945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:12.784903 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn"] Oct 30 18:23:13.632399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:13.631793 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" event={"ID":"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0","Type":"ContainerStarted","Data":"6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930"} Oct 30 18:23:13.632399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:13.631878 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" event={"ID":"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0","Type":"ContainerStarted","Data":"94bce3a27673e73f1c24ea81869cd96c758bd0fbd8de2940d2364fa2980efd2a"} Oct 30 18:23:13.713199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:13.713132 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" podStartSLOduration=1.7131102120000001 podStartE2EDuration="1.713110212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:13.695512239 +0000 UTC m=+1288.084040349" watchObservedRunningTime="2026-10-30 18:23:13.713110212 +0000 UTC m=+1288.101638332" Oct 30 18:23:14.333917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:14.333878 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:14.440770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:14.440740 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:14.442305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:14.442279 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:23:14.442823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:14.442330 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 18:23:15.441002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:15.440970 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:23:15.441841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:15.441022 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 18:23:16.331058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.331019 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df"] Oct 30 18:23:16.340062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.336765 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.340062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.339583 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df"] Oct 30 18:23:16.363745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.359831 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f8709f-serving-cert\"" Oct 30 18:23:16.369989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.369954 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ggjr\" (UniqueName: \"kubernetes.io/projected/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-kube-api-access-2ggjr\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.370122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.370033 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-maas-api-tls\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.370268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.370239 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-metrics-certs\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.370384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.370284 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.441842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.440587 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:23:16.441842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.440633 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 18:23:16.459906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.459875 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:16.472284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.472243 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-metrics-certs\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.472991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.472292 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.472991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.472463 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ggjr\" (UniqueName: \"kubernetes.io/projected/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-kube-api-access-2ggjr\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.472991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.472498 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-maas-api-tls\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.472991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.472921 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-openshift-service-ca\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.479025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.479004 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-maas-api-tls\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.479543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.479059 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-metrics-certs\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.507958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.505628 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ggjr\" (UniqueName: \"kubernetes.io/projected/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-kube-api-access-2ggjr\") pod \"maas-api-e2e-api-a-f8709f-698b768567-kj8df\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:16.595012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.578233 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:16.665339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:16.665304 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:17.078849 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:23:17.078816 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7a7f1d_20bd_41f1_b298_14543a6d2d1d.slice/crio-a1d3cae649db21c72af16958cbf95f0a7835991de6173219c68b9d92691210de WatchSource:0}: Error finding container a1d3cae649db21c72af16958cbf95f0a7835991de6173219c68b9d92691210de: Status 404 returned error can't find the container with id a1d3cae649db21c72af16958cbf95f0a7835991de6173219c68b9d92691210de Oct 30 18:23:17.080633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:17.080610 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df"] Oct 30 18:23:17.440423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:17.440390 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:23:17.440622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:17.440445 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 18:23:17.935367 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:17.935329 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" event={"ID":"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d","Type":"ContainerStarted","Data":"a1d3cae649db21c72af16958cbf95f0a7835991de6173219c68b9d92691210de"} Oct 30 18:23:18.350091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:18.350001 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:18.440817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:18.440378 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 18:23:18.440817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:18.440435 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 18:23:18.942134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:18.942097 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" event={"ID":"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d","Type":"ContainerStarted","Data":"6c02206141a75b56837587431fcd77b4d570aee3741118ee7921da10c1748c95"} Oct 30 18:23:18.942854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:18.942827 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:18.985506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:18.985452 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" podStartSLOduration=2.168257278 podStartE2EDuration="2.98544045s" totalImagesPullingTime="817.183172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:18.984319219 +0000 UTC m=+1293.372847336" watchObservedRunningTime="2026-10-30 18:23:18.98544045 +0000 UTC m=+1293.373968566" Oct 30 18:23:19.445689 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:19.445655 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:19.446138 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:19.446111 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:19.446730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:19.446708 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:23:19.941527 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:23:19.941492 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:23:20.017172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.017139 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:20.119216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.119182 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:20.329871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.329773 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:20.787883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.787843 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm"] Oct 30 18:23:20.794988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.794187 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.818777 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.816491 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm"] Oct 30 18:23:20.818777 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.818743 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-295497-openshift-default-dockercfg-z8bdv\"" Oct 30 18:23:20.828064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.828013 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:20.882809 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.882775 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51cc775-0d90-40a5-9c60-5d802467532a-istiod-ca-cert\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.883007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.882820 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-certs\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.883007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.882843 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-istio-token\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.883139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.883106 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-envoy\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.883226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.883209 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbfjq\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-kube-api-access-cbfjq\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.883373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.883271 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-socket\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.883373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.883315 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51cc775-0d90-40a5-9c60-5d802467532a-istio-podinfo\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.890456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.883387 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-data\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.890456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.883494 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-credential-socket\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.987186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.987120 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfjq\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-kube-api-access-cbfjq\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.987512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.987488 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-socket\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.987679 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.987662 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51cc775-0d90-40a5-9c60-5d802467532a-istio-podinfo\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.987867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.987849 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-data\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.988109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.988089 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-credential-socket\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.988340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.988317 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-socket\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.989020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.988556 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51cc775-0d90-40a5-9c60-5d802467532a-istiod-ca-cert\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.989020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.988586 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-certs\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.989020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.988609 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-istio-token\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.989020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.988700 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-envoy\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.990615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.990594 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-data\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.991212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.991001 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-certs\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.991862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.991760 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51cc775-0d90-40a5-9c60-5d802467532a-istiod-ca-cert\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:20.992639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:20.992619 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-credential-socket\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:21.012697 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.012088 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfjq\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-kube-api-access-cbfjq\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:21.013355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.013234 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-istio-token\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:21.014243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.014217 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:21.014645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.014625 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-envoy\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:21.015215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.015166 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51cc775-0d90-40a5-9c60-5d802467532a-istio-podinfo\") pod \"e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:21.129559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.129532 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:21.466810 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:23:21.466781 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51cc775_0d90_40a5_9c60_5d802467532a.slice/crio-b0837df51f1727871febbd818293c5232472e10cba135404fc7ffa83c054c478 WatchSource:0}: Error finding container b0837df51f1727871febbd818293c5232472e10cba135404fc7ffa83c054c478: Status 404 returned error can't find the container with id b0837df51f1727871febbd818293c5232472e10cba135404fc7ffa83c054c478 Oct 30 18:23:21.469754 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.469723 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:23:21.470509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.469800 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:23:21.470509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.469855 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:23:21.473528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.473482 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm"] Oct 30 18:23:21.633019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:21.632985 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:22.040877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:22.040844 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" event={"ID":"b51cc775-0d90-40a5-9c60-5d802467532a","Type":"ContainerStarted","Data":"1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e"} Oct 30 18:23:22.041568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:22.040895 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" event={"ID":"b51cc775-0d90-40a5-9c60-5d802467532a","Type":"ContainerStarted","Data":"b0837df51f1727871febbd818293c5232472e10cba135404fc7ffa83c054c478"} Oct 30 18:23:22.116458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:22.116405 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" podStartSLOduration=2.116390358 podStartE2EDuration="2.116390358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:22.096560786 +0000 UTC m=+1296.485088905" watchObservedRunningTime="2026-10-30 18:23:22.116390358 +0000 UTC m=+1296.504918521" Oct 30 18:23:22.130187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:22.130155 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:22.137149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:22.137126 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:23.049573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:23.049541 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:23.050679 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:23.050659 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:23:23.202138 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:23.202105 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:24.511880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:24.511845 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:24.609061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:24.609020 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:24.924471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:24.924437 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:24.972468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:24.972442 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:23:25.017657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:25.017626 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:23:26.226762 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.226723 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:26.320828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.320785 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:26.411155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.411121 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5"] Oct 30 18:23:26.416495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.416475 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:26.437652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.437625 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5"] Oct 30 18:23:26.443820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.443794 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-295497\"" Oct 30 18:23:26.488377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.488290 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-295497-dockercfg-6hmv9\"" Oct 30 18:23:26.573599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.573542 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpmkc\" (UniqueName: \"kubernetes.io/projected/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-kube-api-access-wpmkc\") pod \"payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5\" (UID: \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:26.574374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.574342 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5\" (UID: \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:26.668245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.668196 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:26.676353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.676316 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5\" (UID: \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:26.676564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.676392 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpmkc\" (UniqueName: \"kubernetes.io/projected/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-kube-api-access-wpmkc\") pod \"payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5\" (UID: \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:26.677497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.677475 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5\" (UID: \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:26.701679 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.700704 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpmkc\" (UniqueName: \"kubernetes.io/projected/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-kube-api-access-wpmkc\") pod \"payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5\" (UID: \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:26.740006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:26.737399 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:27.096116 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:23:27.095923 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d633c4_c9c8_4b8e_948e_f44f73a6dbd2.slice/crio-d12e82f96cf15235e2119ac7f2b857b4cbc858a61e3cd8dd4abd6f9574dfedda WatchSource:0}: Error finding container d12e82f96cf15235e2119ac7f2b857b4cbc858a61e3cd8dd4abd6f9574dfedda: Status 404 returned error can't find the container with id d12e82f96cf15235e2119ac7f2b857b4cbc858a61e3cd8dd4abd6f9574dfedda Oct 30 18:23:27.099020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:27.098997 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5"] Oct 30 18:23:28.114290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:28.114251 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" event={"ID":"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2","Type":"ContainerStarted","Data":"e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23"} Oct 30 18:23:28.115861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:28.114311 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" event={"ID":"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2","Type":"ContainerStarted","Data":"d12e82f96cf15235e2119ac7f2b857b4cbc858a61e3cd8dd4abd6f9574dfedda"} Oct 30 18:23:28.115861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:28.114358 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:28.161473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:28.161424 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" podStartSLOduration=2.161409135 podStartE2EDuration="2.161409135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:28.154800521 +0000 UTC m=+1302.543328651" watchObservedRunningTime="2026-10-30 18:23:28.161409135 +0000 UTC m=+1302.549937251" Oct 30 18:23:28.279295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:28.279251 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:28.378842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:28.378813 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:28.549636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:28.549594 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:30.423990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:30.422882 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:32.171564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:32.168500 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:32.308967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:32.307952 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:23:32.420402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:32.419736 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:23:32.589013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:32.588888 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:23:32.730089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:32.720006 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:32.730089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:32.720356 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:23:32.767871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:32.759993 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:23:32.785044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:32.785012 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:34.686184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:34.686146 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:36.568038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:36.568002 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:38.231181 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:38.231141 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:39.129524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:39.129489 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:23:41.422170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:41.422136 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:41.726661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:41.726581 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:42.046567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:42.046479 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:43.725200 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:43.725159 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:43.917416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:43.917388 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:44.418545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:44.418510 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:44.601627 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:44.601575 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:45.884160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:45.884119 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:46.260117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:46.260026 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29"] Oct 30 18:23:46.305488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:46.305456 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw"] Oct 30 18:23:46.419528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:46.419467 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:46.520056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:46.519926 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:48.029216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:48.029181 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:48.326226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:48.326129 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:48.442651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:48.442611 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:49.921817 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:23:49.921779 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:23:50.120700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:50.120667 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:50.232440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:50.232344 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:51.261301 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:51.261250 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" containerID="cri-o://312c551f0b54573928369e8cbfa01d9ed6ede085630b13ab90ef3099609a7778" gracePeriod=25 Oct 30 18:23:51.309594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:51.309542 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" containerID="cri-o://b65fba361799f6bdda7a0ddf7df18f305a0289bb95461095c24bab70632802d8" gracePeriod=25 Oct 30 18:23:52.127650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:52.127603 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:52.823571 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:52.823532 5624 scope.go:122] "RemoveContainer" containerID="14a5a749352d09cfa86aa375fc99787efe901ec7978e8450c0c414be183cc511" Oct 30 18:23:52.839602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:52.839577 5624 scope.go:122] "RemoveContainer" containerID="6a4034dc94910b6e8238b469dfda6c23e982a028f1aa00a2f87808d961d20f58" Oct 30 18:23:53.201959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:53.196304 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:53.869873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:53.869835 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:54.003994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:54.000948 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:54.113960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:54.100802 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:54.395019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:54.386889 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:54.432839 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:54.432802 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:23:54.442322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:54.442289 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:23:54.543961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:54.529024 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:55.851331 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:55.851291 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:57.555332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:57.555303 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:58.573782 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:58.573749 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:58.681889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:58.681859 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:59.371139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:23:59.371103 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:00.355568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:00.355535 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:00.455404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:00.455369 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:01.540553 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:01.540517 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:01.660288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:01.660253 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:02.418011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:02.417897 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:02.863488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:02.863403 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:03.384517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:03.384448 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:03.537567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:03.537478 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:04.062119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:04.061680 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:04.204552 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:04.204519 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:04.396020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:04.395923 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:24:04.400752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:04.396044 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:24:04.865172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:04.865070 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:04.865172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:04.865150 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:05.924013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:05.923979 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:07.755395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:07.755353 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:07.858432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:07.858386 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:08.566087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:08.566038 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" podUID="54747d07-9037-4c21-be5c-c554f0b21b5d" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:08.751064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:08.751026 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:09.148545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:09.148503 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 18:24:09.634088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:09.634038 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" podUID="09728775-6be0-45a5-bfb9-480ff7a90759" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:09.679083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:09.679042 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:09.897849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:09.897773 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:10.721066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:10.721034 5624 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-55lt6 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:10.915778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:10.915744 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:11.215076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:11.215040 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" podUID="13b40734-590f-4568-b812-c77303a52d56" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.125:50052\" within 1s: context deadline exceeded" Oct 30 18:24:11.313115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:11.313077 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" podUID="8ca1d631-c3c1-4e4f-910c-76a1310a327c" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:11.591195 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:11.591114 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:12.815425 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:12.815386 5624 logging.go:55] [core] [Channel #1793 SubChannel #1795]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.125:50052", ServerName: "10.134.0.125:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:57638->10.134.0.125:50052: use of closed network connection" Oct 30 18:24:13.130067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:13.130039 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.040515 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:15.104897 5624 logging.go:55] [core] [Channel #1808 SubChannel #1811]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.116:50052: connect: connection refused" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:15.104966 5624 logging.go:55] [core] [Channel #1809 SubChannel #1810]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.117:50052", ServerName: "10.134.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.117:50052: connect: connection refused" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.168087 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx"] Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.168222 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9"] Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.168254 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc"] Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.168443 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" containerID="cri-o://4fc181797cc9cfd2850f57390c80ec7c99d91a064bedd6156b7efcdd77ada93f" gracePeriod=30 Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.169007 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" containerID="cri-o://542dc67a2e64927e5d8153fa9809469740c492de80d3e1756e9586cded3cc97f" gracePeriod=30 Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.169872 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" containerID="cri-o://fe08d835d87bc1ba652ba539b73b240101d4a47824f5a86a29e9b4368124f595" gracePeriod=30 Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:15.175852 5624 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b2832b_135a_4f42_8864_15f49be4c6a6.slice/crio-conmon-312c551f0b54573928369e8cbfa01d9ed6ede085630b13ab90ef3099609a7778.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.410130 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-84jnk" podUID="dc1ecec4-614d-4cbf-978b-7b0fdb293d0f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.902662 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.117:50052\" within 1s: context deadline exceeded" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.902778 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.910201 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.910281 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:15.958056 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" podUID="4326dbaf-621c-4d40-9f86-e29dbdd1b649" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.55:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:16.010242 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" podUID="a1085804-d290-4dfd-8b33-bbbbc3c28009" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.113:8443/health\": context deadline exceeded" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:16.453041 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" podUID="75ee8c7b-b877-4f5c-9b74-08bda044c20c" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:16.530166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:16.492615 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:16.905222 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.61:50052\" within 1s: context deadline exceeded" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:16.907020 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:16.927461 5624 generic.go:356] "Generic (PLEG): container finished" podID="01d378e9-9da2-479b-8856-200a64f66c55" containerID="b65fba361799f6bdda7a0ddf7df18f305a0289bb95461095c24bab70632802d8" exitCode=0 Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:16.927728 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" event={"ID":"01d378e9-9da2-479b-8856-200a64f66c55","Type":"ContainerDied","Data":"b65fba361799f6bdda7a0ddf7df18f305a0289bb95461095c24bab70632802d8"} Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:16.931718 5624 generic.go:356] "Generic (PLEG): container finished" podID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerID="312c551f0b54573928369e8cbfa01d9ed6ede085630b13ab90ef3099609a7778" exitCode=0 Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:17.081455 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:17.341080 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:17.419648 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.080026 5624 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-55lt6 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.080104 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" podUID="8ca1d631-c3c1-4e4f-910c-76a1310a327c" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.138893 5624 patch_prober.go:29] interesting pod/thanos-querier-74f988c6b9-jw4rr container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.138980 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" podUID="269d1132-4f1d-4b95-b545-2b2659d245a7" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.11:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.142689 5624 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-f87jc container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9443/health\": context deadline exceeded" start-of-body= Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.142750 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" podUID="9e62ca62-c183-4859-8d52-a52aa8182c73" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.12:9443/health\": context deadline exceeded" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.364967 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.393040 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" podUID="92cd679c-b797-4195-aae4-5312581fbce9" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.512801 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.583966 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-r8j74" podUID="54747d07-9037-4c21-be5c-c554f0b21b5d" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.703835 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.706672 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:24:19.474569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:18.706724 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:24:24.866019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:19.558206 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:24.866019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:19.562651 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:24:24.866019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:19.563397 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:24.866019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:19.567311 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:24.866019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:19.568754 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:24:24.866019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:19.599855 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 18:24:32.821140 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:32.821102 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:32.822734 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" podUID="a1085804-d290-4dfd-8b33-bbbbc3c28009" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.113:8443/health\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:32.842439 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:32.981598 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" podUID="8c0ef545-bf5b-4665-95a8-2c6b578138a3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.105:50052: i/o timeout" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.032555 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.084345 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.087473 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.089597 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" podUID="2d81034f-ac92-46ff-a8d6-a432835b1426" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.106:50052: i/o timeout" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.093488 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.095867 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" podUID="f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.130:50052\" within 1s: context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.096280 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" podUID="0ffdb840-fd18-4ad2-8635-2234ab2069b0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.124:50052\" within 1s: context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.097904 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.62:50052\" within 1s: context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.099283 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-769875db56-84jnk" podUID="dc1ecec4-614d-4cbf-978b-7b0fdb293d0f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.38:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.099325 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" podUID="0a09133a-00c6-4632-ba5e-b859f8da282e" containerName="server" probeResult="failure" output="Get \"https://10.134.0.37:8443/healthz\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.099568 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" event={"ID":"f5b2832b-135a-4f42-8864-15f49be4c6a6","Type":"ContainerDied","Data":"312c551f0b54573928369e8cbfa01d9ed6ede085630b13ab90ef3099609a7778"} Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.228350 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.228442 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.228505 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.228546 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.230220 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" podUID="60543fc7-02a7-403a-bb27-bd46cd63591a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.118:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.233591 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.238046 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.237921 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.243574 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" podUID="c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.65:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.245872 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-limitador-7c9b87d574-2hj6q" podUID="5c4422bc-c3e1-48f1-99a8-c62260e1d25c" containerName="limitador" probeResult="failure" output="Get \"http://10.134.0.36:8080/status\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:32.626143 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.256486 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:32.627695 5624 patch_prober.go:29] interesting pod/thanos-querier-74f988c6b9-jw4rr container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:9091/-/ready\": context deadline exceeded" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.256549 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" podUID="269d1132-4f1d-4b95-b545-2b2659d245a7" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.11:9091/-/ready\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.083824 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.256600 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.087267 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.256630 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:33.096595 5624 logging.go:55] [core] [Channel #1825 SubChannel #1834]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.130:50052", ServerName: "10.134.0.130:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.130:50052: operation was canceled" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:33.096752 5624 logging.go:55] [core] [Channel #1832 SubChannel #1836]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.124:50052", ServerName: "10.134.0.124:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.124:50052: operation was canceled" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:33.098105 5624 logging.go:55] [core] [Channel #1824 SubChannel #1841]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.62:50052", ServerName: "10.134.0.62:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.62:50052: operation was canceled" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.206791 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:33.209832 5624 logging.go:55] [core] [Channel #1826 SubChannel #1833]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.117:50052", ServerName: "10.134.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.117:50052: connect: connection refused" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.256792 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.269008 5624 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-55lt6 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.269050 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" podUID="8ca1d631-c3c1-4e4f-910c-76a1310a327c" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.281115 5624 patch_prober.go:29] interesting pod/dns-default-6mcgq container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.281153 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-6mcgq" podUID="b1cd5923-8633-4b24-8310-e06ac0a2de68" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.490441 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.117:50052\" within 1s: context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.492538 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.61:50052\" within 1s: context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.493521 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.535097 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" podUID="92cd679c-b797-4195-aae4-5312581fbce9" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.536094 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" podUID="75ee8c7b-b877-4f5c-9b74-08bda044c20c" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:33.537292 5624 logging.go:55] [core] [Channel #1827 SubChannel #1835]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.61:50052", ServerName: "10.134.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:60678->10.134.0.61:50052: use of closed network connection" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.603618 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.603675 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:33.607656 5624 logging.go:55] [core] [Channel #1877 SubChannel #1878]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.116:50052: connect: connection refused" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.626073 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.705583 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.705623 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 18:24:33.817945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.713828 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.864082 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" podUID="92cd679c-b797-4195-aae4-5312581fbce9" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:33.998147 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.053148 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" podUID="4326dbaf-621c-4d40-9f86-e29dbdd1b649" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.55:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.085712 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.086184 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.089060 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.103070 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" podUID="0a09133a-00c6-4632-ba5e-b859f8da282e" containerName="server" probeResult="failure" output="Get \"https://10.134.0.37:8443/healthz\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.136122 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.266559 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.290634267s: [/var/lib/containers/storage/overlay/a338c7dc4d10ae085daf7c70743cbe80563a8cdd4bee85443501bfa9b99dc761/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5_f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:34.394728 5624 logging.go:55] [core] [Channel #1879 SubChannel #1880]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.117:50052", ServerName: "10.134.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.117:50052: connect: connection refused" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.569401 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.570286 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.571140 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.571173 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573153 5624 patch_prober.go:29] interesting pod/thanos-querier-74f988c6b9-jw4rr container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:9091/-/ready\": context deadline exceeded" start-of-body= Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573178 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" podUID="0a09133a-00c6-4632-ba5e-b859f8da282e" containerName="server" probeResult="failure" output="Get \"https://10.134.0.37:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573193 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" podUID="269d1132-4f1d-4b95-b545-2b2659d245a7" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.11:9091/-/ready\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573188 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573235 5624 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-f87jc container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.12:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573252 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573263 5624 patch_prober.go:29] interesting pod/thanos-querier-74f988c6b9-jw4rr container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.11:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573185 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573285 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-74f988c6b9-jw4rr" podUID="269d1132-4f1d-4b95-b545-2b2659d245a7" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.11:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573264 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-f87jc" podUID="9e62ca62-c183-4859-8d52-a52aa8182c73" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.12:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.573314 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.575070 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.575086 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.575118 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.575725 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.575763 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.575827 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.575970 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.576760 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.64:50052\" within 1s: context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.600221 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.605867 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.606581 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.606711 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:34.607163 5624 logging.go:55] [core] [Channel #1884 SubChannel #1885]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.116:50052: connect: connection refused" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.616059 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" podUID="09728775-6be0-45a5-bfb9-480ff7a90759" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.616129 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:34.617141 5624 logging.go:55] [core] [Channel #1861 SubChannel #1862]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.64:50052", ServerName: "10.134.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39300->10.134.0.64:50052: use of closed network connection" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.618060 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" podUID="09728775-6be0-45a5-bfb9-480ff7a90759" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.623050 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" podUID="4326dbaf-621c-4d40-9f86-e29dbdd1b649" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.55:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.627231 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.653478 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-84jnk" podUID="dc1ecec4-614d-4cbf-978b-7b0fdb293d0f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.655012 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" podUID="75ee8c7b-b877-4f5c-9b74-08bda044c20c" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.714880 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.824058 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:34.824111 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.086258 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" podUID="d2333eaa-171b-4dd4-a978-24ba23defed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.086294 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.089888 5624 trace.go:236] Trace[1403984503]: "Calculate volume metrics of kube-state-metrics-custom-resource-state-configmap for pod openshift-monitoring/kube-state-metrics-5dc8d7d89b-brcll" (30-Oct-2026 18:24:34.012) (total time: 1077ms): Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: Trace[1403984503]: [1.077136904s] [1.077136904s] END Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.089973 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" podUID="e069f9a5-f97d-45a9-a64a-d47a4118f251" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.269029 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" podUID="4326dbaf-621c-4d40-9f86-e29dbdd1b649" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.55:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.342293 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" podUID="8fef5526-096c-45de-a569-4e80570d9b28" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.60:8443/health\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.394557 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.117:50052\" within 1s: context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.569813 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.571156 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" podUID="32a32196-f4a2-41c3-b028-25e792821d05" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.573524 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-49l6m" podUID="0a09133a-00c6-4632-ba5e-b859f8da282e" containerName="server" probeResult="failure" output="Get \"https://10.134.0.37:8443/readyz\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.573554 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.573704 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.576059 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.576071 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.576088 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" podUID="79e88ba7-0418-4268-b1b3-e50946e634eb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.576118 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.576093 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.576922 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.64:50052\" within 1s: context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.599148 5624 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.370016852s: [/var/lib/containers/storage/overlay/f46c87bee84aaa64bc53992c32d85d222309a4c276d76c53dbd09efca2bf8475/diff /var/log/pods/openshift-ingress_e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm_b51cc775-0d90-40a5-9c60-5d802467532a/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.600684 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.606972 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:35.618092 5624 logging.go:55] [core] [Channel #1881 SubChannel #1882]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.64:50052", ServerName: "10.134.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:39314->10.134.0.64:50052: use of closed network connection" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.659035 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" podUID="09728775-6be0-45a5-bfb9-480ff7a90759" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.723021 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.728027 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-84jnk" podUID="dc1ecec4-614d-4cbf-978b-7b0fdb293d0f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.777983 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" podUID="eba05bdf-704e-4dc8-9da9-3e52b6f0a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.806094 5624 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-55lt6 container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.806133 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" podUID="8ca1d631-c3c1-4e4f-910c-76a1310a327c" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:35.826576 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:36.244314 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" podUID="4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.128:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:36.453060 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" podUID="75ee8c7b-b877-4f5c-9b74-08bda044c20c" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:36.569625 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" podUID="60543fc7-02a7-403a-bb27-bd46cd63591a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.118:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:36.571004 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" podUID="a1085804-d290-4dfd-8b33-bbbbc3c28009" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.113:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:36.571015 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" podUID="c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.65:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:36.573736 5624 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-55lt6 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:36.573782 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" podUID="8ca1d631-c3c1-4e4f-910c-76a1310a327c" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:37.163106 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-47.ec2.internal" podUID="05806b328c870eba3d65f2b11ca3bf01" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:37.343038 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:37.414100 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:37.414151 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:37.905663 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-g74ls" podUID="82127ba1-4bef-43cd-8550-719a4402fcab" containerName="sbdb" probeResult="failure" output="command timed out" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.096642 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:38.357785 5624 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="5.258s" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.372664 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.372706 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd"] Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.372762 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.372951 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.373133 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.373171 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.373209 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.373248 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.373341 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:24:38.379202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.373530 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380269 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380317 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380355 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380398 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380429 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380463 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-7wj9m" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380537 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380584 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380618 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380647 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.380733 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.393817 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.33:8081/readyz\": dial tcp 10.134.0.33:8081: connect: connection refused" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394374 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394482 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-lz5rk" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394512 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7gjfw" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394571 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394595 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394616 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394633 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394663 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394676 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394701 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394742 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.394786 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.395560 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kl2vt" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:38.455312 5624 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-serving-cert: secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:38.455384 5624 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls podName:60543fc7-02a7-403a-bb27-bd46cd63591a nodeName:}" failed. No retries permitted until 2026-10-30 18:24:38.955357752 +0000 UTC m=+1373.343885865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls") pod "maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" (UID: "60543fc7-02a7-403a-bb27-bd46cd63591a") : secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:38.489642 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-84jnk" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:38.970669 5624 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-serving-cert: secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:38.970738 5624 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls podName:60543fc7-02a7-403a-bb27-bd46cd63591a nodeName:}" failed. No retries permitted until 2026-10-30 18:24:39.970718837 +0000 UTC m=+1374.359246935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls") pod "maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" (UID: "60543fc7-02a7-403a-bb27-bd46cd63591a") : secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.326638 5624 generic.go:356] "Generic (PLEG): container finished" podID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerID="542dc67a2e64927e5d8153fa9809469740c492de80d3e1756e9586cded3cc97f" exitCode=0 Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.326754 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" event={"ID":"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3","Type":"ContainerDied","Data":"542dc67a2e64927e5d8153fa9809469740c492de80d3e1756e9586cded3cc97f"} Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.455401 5624 generic.go:356] "Generic (PLEG): container finished" podID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerID="4fc181797cc9cfd2850f57390c80ec7c99d91a064bedd6156b7efcdd77ada93f" exitCode=0 Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.455664 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" event={"ID":"0ee5582b-6f2b-41b6-82df-39fa47df1265","Type":"ContainerDied","Data":"4fc181797cc9cfd2850f57390c80ec7c99d91a064bedd6156b7efcdd77ada93f"} Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.459664 5624 generic.go:356] "Generic (PLEG): container finished" podID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerID="fe08d835d87bc1ba652ba539b73b240101d4a47824f5a86a29e9b4368124f595" exitCode=0 Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.464493 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" event={"ID":"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce","Type":"ContainerDied","Data":"fe08d835d87bc1ba652ba539b73b240101d4a47824f5a86a29e9b4368124f595"} Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.466297 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" podUID="60543fc7-02a7-403a-bb27-bd46cd63591a" containerName="maas-api" containerID="cri-o://38441a79b7ace0de8d22eb18626f1082a6f1554e678a1d8774c44e02fa59428e" gracePeriod=30 Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.472686 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5bcd7466d5-kzdwm" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:39.835458 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:39.889029 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:40.062042 5624 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-serving-cert: secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:40.062154 5624 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls podName:60543fc7-02a7-403a-bb27-bd46cd63591a nodeName:}" failed. No retries permitted until 2026-10-30 18:24:42.062137439 +0000 UTC m=+1376.450665538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls") pod "maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" (UID: "60543fc7-02a7-403a-bb27-bd46cd63591a") : secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.224519 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.224585 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.482965 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" event={"ID":"60543fc7-02a7-403a-bb27-bd46cd63591a","Type":"ContainerDied","Data":"38441a79b7ace0de8d22eb18626f1082a6f1554e678a1d8774c44e02fa59428e"} Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.483080 5624 generic.go:356] "Generic (PLEG): container finished" podID="60543fc7-02a7-403a-bb27-bd46cd63591a" containerID="38441a79b7ace0de8d22eb18626f1082a6f1554e678a1d8774c44e02fa59428e" exitCode=0 Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.494337 5624 generic.go:356] "Generic (PLEG): container finished" podID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerID="695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a" exitCode=0 Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.494624 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" event={"ID":"bd4c541a-41f4-4af4-a006-3ce2155611bd","Type":"ContainerDied","Data":"695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a"} Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.495177 5624 scope.go:122] "RemoveContainer" containerID="695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a" Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.500987 5624 generic.go:356] "Generic (PLEG): container finished" podID="09728775-6be0-45a5-bfb9-480ff7a90759" containerID="5d3918195112a63cf2e8bbe905f64a986bd507845a3025275dda8bfabb3e8d27" exitCode=1 Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.503207 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" event={"ID":"09728775-6be0-45a5-bfb9-480ff7a90759","Type":"ContainerDied","Data":"5d3918195112a63cf2e8bbe905f64a986bd507845a3025275dda8bfabb3e8d27"} Oct 30 18:24:40.676263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.504239 5624 scope.go:122] "RemoveContainer" containerID="5d3918195112a63cf2e8bbe905f64a986bd507845a3025275dda8bfabb3e8d27" Oct 30 18:24:41.208502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.721034 5624 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-55lt6 container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:24:41.208502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.721226 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-55lt6" podUID="8ca1d631-c3c1-4e4f-910c-76a1310a327c" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:24:41.208502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.727306 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65"] Oct 30 18:24:41.208502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.727481 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" containerID="cri-o://850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273" gracePeriod=30 Oct 30 18:24:41.208502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.840504 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g"] Oct 30 18:24:41.208502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:40.840715 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" containerID="cri-o://ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c" gracePeriod=30 Oct 30 18:24:41.208502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:41.062831 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz"] Oct 30 18:24:41.208502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:41.063039 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" containerID="cri-o://78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608" gracePeriod=30 Oct 30 18:24:41.327597 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:41.226761 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" podUID="60543fc7-02a7-403a-bb27-bd46cd63591a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.118:8443/health\": dial tcp 10.134.0.118:8443: connect: connection refused" Oct 30 18:24:41.400783 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:41.375696 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:41.674039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:41.532574 5624 trace.go:236] Trace[1787186262]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-s7xqz" (30-Oct-2026 18:24:38.698) (total time: 2834ms): Oct 30 18:24:41.674039 ip-10-0-133-47 kubenswrapper[5624]: Trace[1787186262]: [2.834103219s] [2.834103219s] END Oct 30 18:24:41.674039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:41.571178 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:24:41.843802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:41.843774 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:42.126945 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:42.126913 5624 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-serving-cert: secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:24:42.127138 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:42.127004 5624 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls podName:60543fc7-02a7-403a-bb27-bd46cd63591a nodeName:}" failed. No retries permitted until 2026-10-30 18:24:46.126989312 +0000 UTC m=+1380.515517410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls") pod "maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" (UID: "60543fc7-02a7-403a-bb27-bd46cd63591a") : secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:24:43.279271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.279247 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:24:43.287104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.287062 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:43.301004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.300967 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-kube-api-access-c7k5g\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-kube-api-access-c7k5g\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.301142 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.301049 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-credential-socket\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.301903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.301178 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-certs\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.301903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.301286 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-podinfo\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.301903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.301389 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istiod-ca-cert\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.301903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.301448 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-token\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-token\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.301903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.301487 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-socket\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.301903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.301689 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-envoy\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.301903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.301729 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-data\") pod \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\" (UID: \"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce\") " Oct 30 18:24:43.303090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.302245 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:24:43.311471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.311448 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-certs" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.311889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.311872 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istiod-ca-cert" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.313270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.313206 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-credential-socket" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.313346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.313269 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-socket" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.313797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.313777 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-data" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.336090 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.336070 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:24:43.343794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.343718 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:24:43.358686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.358663 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:24:43.371579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.371555 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:24:43.393641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.393583 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-token" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.399208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.399183 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-envoy" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.405388 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-podinfo" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.405469 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-kube-api-access-c7k5g" pod "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" (UID: "4f10c059-42a2-4ac2-b8bf-0a7aee7061ce"). InnerVolumeSpecName "kube-api-access-c7k5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.405927 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-podinfo\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406056 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406094 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-token\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-token\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406124 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-envoy\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406164 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-kube-api-access-zxc74\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-kube-api-access-zxc74\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406203 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls\") pod \"60543fc7-02a7-403a-bb27-bd46cd63591a\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406226 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-socket\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406280 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-socket\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406334 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-credential-socket\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406395 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b2832b-135a-4f42-8864-15f49be4c6a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b2832b-135a-4f42-8864-15f49be4c6a6-plugins-config-volume\") pod \"f5b2832b-135a-4f42-8864-15f49be4c6a6\" (UID: \"f5b2832b-135a-4f42-8864-15f49be4c6a6\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406438 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-envoy\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406467 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d378e9-9da2-479b-8856-200a64f66c55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d378e9-9da2-479b-8856-200a64f66c55-plugins-config-volume\") pod \"01d378e9-9da2-479b-8856-200a64f66c55\" (UID: \"01d378e9-9da2-479b-8856-200a64f66c55\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406509 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-certs\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406528 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-token\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-token\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406566 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60543fc7-02a7-403a-bb27-bd46cd63591a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60543fc7-02a7-403a-bb27-bd46cd63591a-openshift-service-ca\") pod \"60543fc7-02a7-403a-bb27-bd46cd63591a\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406604 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-podinfo\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406627 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-kube-api-access-cpjxd\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-kube-api-access-cpjxd\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406709 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-metrics-certs\") pod \"60543fc7-02a7-403a-bb27-bd46cd63591a\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406766 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istiod-ca-cert\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406838 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b2832b-135a-4f42-8864-15f49be4c6a6-kube-api-access-tk6f9\" (UniqueName: \"kubernetes.io/projected/f5b2832b-135a-4f42-8864-15f49be4c6a6-kube-api-access-tk6f9\") pod \"f5b2832b-135a-4f42-8864-15f49be4c6a6\" (UID: \"f5b2832b-135a-4f42-8864-15f49be4c6a6\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406865 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60543fc7-02a7-403a-bb27-bd46cd63591a-kube-api-access-xnn8j\" (UniqueName: \"kubernetes.io/projected/60543fc7-02a7-403a-bb27-bd46cd63591a-kube-api-access-xnn8j\") pod \"60543fc7-02a7-403a-bb27-bd46cd63591a\" (UID: \"60543fc7-02a7-403a-bb27-bd46cd63591a\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.406925 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-credential-socket\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.407028 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-data\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.407111 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d378e9-9da2-479b-8856-200a64f66c55-kube-api-access-dtd68\" (UniqueName: \"kubernetes.io/projected/01d378e9-9da2-479b-8856-200a64f66c55-kube-api-access-dtd68\") pod \"01d378e9-9da2-479b-8856-200a64f66c55\" (UID: \"01d378e9-9da2-479b-8856-200a64f66c55\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.407245 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee5582b-6f2b-41b6-82df-39fa47df1265-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee5582b-6f2b-41b6-82df-39fa47df1265-istiod-ca-cert\") pod \"0ee5582b-6f2b-41b6-82df-39fa47df1265\" (UID: \"0ee5582b-6f2b-41b6-82df-39fa47df1265\") " Oct 30 18:24:43.407892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.407302 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-data\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.408465 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d378e9-9da2-479b-8856-200a64f66c55-plugins-config-volume" pod "01d378e9-9da2-479b-8856-200a64f66c55" (UID: "01d378e9-9da2-479b-8856-200a64f66c55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410160 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7k5g\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-kube-api-access-c7k5g\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410182 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410197 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410211 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410223 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410236 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01d378e9-9da2-479b-8856-200a64f66c55-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410248 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410261 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410274 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.410283 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.439108 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-data" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.484040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.444487 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60543fc7-02a7-403a-bb27-bd46cd63591a-kube-api-access-xnn8j" pod "60543fc7-02a7-403a-bb27-bd46cd63591a" (UID: "60543fc7-02a7-403a-bb27-bd46cd63591a"). InnerVolumeSpecName "kube-api-access-xnn8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.497620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.495135 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d378e9-9da2-479b-8856-200a64f66c55-kube-api-access-dtd68" pod "01d378e9-9da2-479b-8856-200a64f66c55" (UID: "01d378e9-9da2-479b-8856-200a64f66c55"). InnerVolumeSpecName "kube-api-access-dtd68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.497620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.496550 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b2832b-135a-4f42-8864-15f49be4c6a6-kube-api-access-tk6f9" pod "f5b2832b-135a-4f42-8864-15f49be4c6a6" (UID: "f5b2832b-135a-4f42-8864-15f49be4c6a6"). InnerVolumeSpecName "kube-api-access-tk6f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.506522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.498396 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b2832b-135a-4f42-8864-15f49be4c6a6-plugins-config-volume" pod "f5b2832b-135a-4f42-8864-15f49be4c6a6" (UID: "f5b2832b-135a-4f42-8864-15f49be4c6a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.514435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.514322 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtd68\" (UniqueName: \"kubernetes.io/projected/01d378e9-9da2-479b-8856-200a64f66c55-kube-api-access-dtd68\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.514435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.514361 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.514435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.514376 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b2832b-135a-4f42-8864-15f49be4c6a6-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.514435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.514391 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk6f9\" (UniqueName: \"kubernetes.io/projected/f5b2832b-135a-4f42-8864-15f49be4c6a6-kube-api-access-tk6f9\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.514435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.514405 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnn8j\" (UniqueName: \"kubernetes.io/projected/60543fc7-02a7-403a-bb27-bd46cd63591a-kube-api-access-xnn8j\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.517490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.517439 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls" pod "60543fc7-02a7-403a-bb27-bd46cd63591a" (UID: "60543fc7-02a7-403a-bb27-bd46cd63591a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:43.524681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.524625 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-metrics-certs" pod "60543fc7-02a7-403a-bb27-bd46cd63591a" (UID: "60543fc7-02a7-403a-bb27-bd46cd63591a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:43.541783 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.540521 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60543fc7-02a7-403a-bb27-bd46cd63591a-openshift-service-ca" pod "60543fc7-02a7-403a-bb27-bd46cd63591a" (UID: "60543fc7-02a7-403a-bb27-bd46cd63591a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.579225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.576693 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" event={"ID":"01d378e9-9da2-479b-8856-200a64f66c55","Type":"ContainerDied","Data":"0d10994020760797631469cd7f5e0cf95ca2b7a7161afee3da160e355aac70b2"} Oct 30 18:24:43.579225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.576770 5624 scope.go:122] "RemoveContainer" containerID="b65fba361799f6bdda7a0ddf7df18f305a0289bb95461095c24bab70632802d8" Oct 30 18:24:43.579225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.576954 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.583827 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" event={"ID":"4f10c059-42a2-4ac2-b8bf-0a7aee7061ce","Type":"ContainerDied","Data":"8a1cf8deb021f211ba3499ebbca7a2b7202f07ee17e2b69951622f1608384fa0"} Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.583929 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.586997 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-kube-api-access-zxc74" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "kube-api-access-zxc74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.609002 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" event={"ID":"09728775-6be0-45a5-bfb9-480ff7a90759","Type":"ContainerStarted","Data":"b9fdbfaa7d2eafd2760dff48a58c77c631f55c166e35373677ea69f3fb372118"} Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.610072 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.617055 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-socket" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.617327 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-credential-socket" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.618003 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.618169 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs\") pod \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\" (UID: \"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3\") " Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.619807 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.619832 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxc74\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-kube-api-access-zxc74\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.619847 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.619864 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.619878 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60543fc7-02a7-403a-bb27-bd46cd63591a-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.619890 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60543fc7-02a7-403a-bb27-bd46cd63591a-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:43.619972 5624 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3/volumes/kubernetes.io~empty-dir/workload-certs Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.619982 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.620167 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istiod-ca-cert" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.620753 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-data" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.623278 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-podinfo" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.624051 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" event={"ID":"60543fc7-02a7-403a-bb27-bd46cd63591a","Type":"ContainerDied","Data":"eefaf28a019c143396792100e7790524f920e64069a2e27850051cb9b6db7462"} Oct 30 18:24:43.626171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.624165 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd" Oct 30 18:24:43.634019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.626760 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-kube-api-access-cpjxd" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "kube-api-access-cpjxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.650910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.644873 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" event={"ID":"f5b2832b-135a-4f42-8864-15f49be4c6a6","Type":"ContainerDied","Data":"f4ef12051c1377544f671f2847930ce3186428499e91baed43b42fffcff2b5fb"} Oct 30 18:24:43.650910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.645002 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29" Oct 30 18:24:43.650910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.649113 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee5582b-6f2b-41b6-82df-39fa47df1265-istiod-ca-cert" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:43.664393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.664249 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-envoy" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.664393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.664362 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-token" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.684440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.684403 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" event={"ID":"bd4c541a-41f4-4af4-a006-3ce2155611bd","Type":"ContainerStarted","Data":"c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54"} Oct 30 18:24:43.685786 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.685759 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:24:43.688059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.688034 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" event={"ID":"c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3","Type":"ContainerDied","Data":"cc0616140386cceb1fde1ed50db245d5f4b8f5cc1c31f23b9d50bfcd7937752f"} Oct 30 18:24:43.688299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.688280 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9" Oct 30 18:24:43.690493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.690472 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" event={"ID":"0ee5582b-6f2b-41b6-82df-39fa47df1265","Type":"ContainerDied","Data":"43d1480605ddc2a55a036850d095cf1599eb4696e9eaeede751ad84d89fcdd02"} Oct 30 18:24:43.690707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.690689 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx" Oct 30 18:24:43.703449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.700492 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-credential-socket" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.703449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.703249 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-certs" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.703449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.703293 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-socket" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734445 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734480 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpjxd\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-kube-api-access-cpjxd\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734497 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734511 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734524 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ee5582b-6f2b-41b6-82df-39fa47df1265-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734543 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734558 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734570 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734583 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734602 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.738237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.734617 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.751094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.751060 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-envoy" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:43.782080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.782042 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-podinfo" pod "0ee5582b-6f2b-41b6-82df-39fa47df1265" (UID: "0ee5582b-6f2b-41b6-82df-39fa47df1265"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:43.782260 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.782175 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-token" pod "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" (UID: "c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:43.856826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.836260 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.856826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.836281 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.856826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.836290 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ee5582b-6f2b-41b6-82df-39fa47df1265-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:43.947521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.919156 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd"] Oct 30 18:24:43.947521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.924773 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-8sfkd"] Oct 30 18:24:43.960205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.960173 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc"] Oct 30 18:24:43.977050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:43.977016 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-brsmc"] Oct 30 18:24:44.021799 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.021766 5624 scope.go:122] "RemoveContainer" containerID="fe08d835d87bc1ba652ba539b73b240101d4a47824f5a86a29e9b4368124f595" Oct 30 18:24:44.053560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.053426 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29"] Oct 30 18:24:44.060133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.060109 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-tfr29"] Oct 30 18:24:44.090928 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.090907 5624 scope.go:122] "RemoveContainer" containerID="38441a79b7ace0de8d22eb18626f1082a6f1554e678a1d8774c44e02fa59428e" Oct 30 18:24:44.102789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.102763 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw"] Oct 30 18:24:44.111012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.110988 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-5ksrw"] Oct 30 18:24:44.136144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.136126 5624 scope.go:122] "RemoveContainer" containerID="312c551f0b54573928369e8cbfa01d9ed6ede085630b13ab90ef3099609a7778" Oct 30 18:24:44.150495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.150475 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx"] Oct 30 18:24:44.156243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.156216 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-blpmx"] Oct 30 18:24:44.164403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.164384 5624 scope.go:122] "RemoveContainer" containerID="542dc67a2e64927e5d8153fa9809469740c492de80d3e1756e9586cded3cc97f" Oct 30 18:24:44.191148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.191124 5624 scope.go:122] "RemoveContainer" containerID="4fc181797cc9cfd2850f57390c80ec7c99d91a064bedd6156b7efcdd77ada93f" Oct 30 18:24:44.198602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.198580 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9"] Oct 30 18:24:44.212024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.212004 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-4xzf9"] Oct 30 18:24:44.971247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:44.971215 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.077586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.077552 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.411297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.411270 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.515755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.515719 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.541598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.541567 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv"] Oct 30 18:24:45.542450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542433 5624 cpu_manager.go:414] "containerMap: removing container" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542451 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542489 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542494 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542503 5624 cpu_manager.go:414] "containerMap: removing container" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542508 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542523 5624 cpu_manager.go:414] "containerMap: removing container" podUID="60543fc7-02a7-403a-bb27-bd46cd63591a" containerName="maas-api" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542528 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60543fc7-02a7-403a-bb27-bd46cd63591a" containerName="maas-api" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542546 5624 cpu_manager.go:414] "containerMap: removing container" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542552 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542562 5624 cpu_manager.go:414] "containerMap: removing container" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542566 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542681 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" containerName="istio-proxy" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542688 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d378e9-9da2-479b-8856-200a64f66c55" containerName="payload-pre-processing" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542696 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="60543fc7-02a7-403a-bb27-bd46cd63591a" containerName="maas-api" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542708 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" containerName="payload-processing" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542714 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" containerName="istio-proxy" Oct 30 18:24:45.551208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.542723 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" containerName="istio-proxy" Oct 30 18:24:45.564781 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.564760 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:45.583665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.583641 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-550276\"" Oct 30 18:24:45.602797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.602775 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786"] Oct 30 18:24:45.623524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.623497 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:45.639440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.639404 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-550276-dockercfg-qrbb8\"" Oct 30 18:24:45.697414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.679327 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2r9g\" (UniqueName: \"kubernetes.io/projected/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-kube-api-access-j2r9g\") pod \"payload-processing-e2e-api-b-550276-94746b766-wzfqv\" (UID: \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\") " pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:45.697414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.679653 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5lhh\" (UniqueName: \"kubernetes.io/projected/4e202b77-f27a-4c86-84b0-ab8cee55c377-kube-api-access-j5lhh\") pod \"payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786\" (UID: \"4e202b77-f27a-4c86-84b0-ab8cee55c377\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:45.697414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.679751 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-550276-94746b766-wzfqv\" (UID: \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\") " pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:45.697414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.679826 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e202b77-f27a-4c86-84b0-ab8cee55c377-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786\" (UID: \"4e202b77-f27a-4c86-84b0-ab8cee55c377\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:45.782341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.781523 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2r9g\" (UniqueName: \"kubernetes.io/projected/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-kube-api-access-j2r9g\") pod \"payload-processing-e2e-api-b-550276-94746b766-wzfqv\" (UID: \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\") " pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:45.782341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.781764 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5lhh\" (UniqueName: \"kubernetes.io/projected/4e202b77-f27a-4c86-84b0-ab8cee55c377-kube-api-access-j5lhh\") pod \"payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786\" (UID: \"4e202b77-f27a-4c86-84b0-ab8cee55c377\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:45.782341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.781848 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-550276-94746b766-wzfqv\" (UID: \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\") " pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:45.782341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.781917 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e202b77-f27a-4c86-84b0-ab8cee55c377-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786\" (UID: \"4e202b77-f27a-4c86-84b0-ab8cee55c377\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:45.791880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.791849 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e202b77-f27a-4c86-84b0-ab8cee55c377-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786\" (UID: \"4e202b77-f27a-4c86-84b0-ab8cee55c377\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:45.806923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:45.806890 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-550276-94746b766-wzfqv\" (UID: \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\") " pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:46.075322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.012183 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5lhh\" (UniqueName: \"kubernetes.io/projected/4e202b77-f27a-4c86-84b0-ab8cee55c377-kube-api-access-j5lhh\") pod \"payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786\" (UID: \"4e202b77-f27a-4c86-84b0-ab8cee55c377\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:46.106871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.106841 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2r9g\" (UniqueName: \"kubernetes.io/projected/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-kube-api-access-j2r9g\") pod \"payload-processing-e2e-api-b-550276-94746b766-wzfqv\" (UID: \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\") " pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:46.186087 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.186055 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d378e9-9da2-479b-8856-200a64f66c55" path="/var/lib/kubelet/pods/01d378e9-9da2-479b-8856-200a64f66c55/volumes" Oct 30 18:24:46.192802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.192775 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee5582b-6f2b-41b6-82df-39fa47df1265" path="/var/lib/kubelet/pods/0ee5582b-6f2b-41b6-82df-39fa47df1265/volumes" Oct 30 18:24:46.193835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.193808 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f10c059-42a2-4ac2-b8bf-0a7aee7061ce" path="/var/lib/kubelet/pods/4f10c059-42a2-4ac2-b8bf-0a7aee7061ce/volumes" Oct 30 18:24:46.197864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.197839 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:46.199764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.199741 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60543fc7-02a7-403a-bb27-bd46cd63591a" path="/var/lib/kubelet/pods/60543fc7-02a7-403a-bb27-bd46cd63591a/volumes" Oct 30 18:24:46.200779 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.200757 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3" path="/var/lib/kubelet/pods/c6dbd6a2-df14-4e02-8e23-8ff769a8d6f3/volumes" Oct 30 18:24:46.211162 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.211139 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b2832b-135a-4f42-8864-15f49be4c6a6" path="/var/lib/kubelet/pods/f5b2832b-135a-4f42-8864-15f49be4c6a6/volumes" Oct 30 18:24:46.211850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.211726 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv"] Oct 30 18:24:46.211850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.211786 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786"] Oct 30 18:24:46.250600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.250375 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:46.492266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.492234 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:24:46.543238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.542948 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0c98c9-ca4c-430f-a465-edc7b68da522-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0c98c9-ca4c-430f-a465-edc7b68da522-istiod-ca-cert\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.543238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.542999 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-kube-api-access-vqf6g\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-kube-api-access-vqf6g\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.543238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.543202 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-token\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-token\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.543238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.543237 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-certs\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.545119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.543327 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-socket\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.545119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.543353 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-credential-socket\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.545119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.543393 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-data\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.545119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.543484 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-podinfo\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.545119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.543591 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-envoy\") pod \"1d0c98c9-ca4c-430f-a465-edc7b68da522\" (UID: \"1d0c98c9-ca4c-430f-a465-edc7b68da522\") " Oct 30 18:24:46.567143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.545724 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-certs" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.567143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.545766 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-credential-socket" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.567143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.546369 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-data" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.567143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.546950 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-socket" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.567143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.548425 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0c98c9-ca4c-430f-a465-edc7b68da522-istiod-ca-cert" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:46.567143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.552443 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-envoy" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.595849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.595803 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-token" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.613917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.599232 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:24:46.613917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.610271 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-podinfo" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:46.613917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.612347 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-kube-api-access-vqf6g" pod "1d0c98c9-ca4c-430f-a465-edc7b68da522" (UID: "1d0c98c9-ca4c-430f-a465-edc7b68da522"). InnerVolumeSpecName "kube-api-access-vqf6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.647907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.647580 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-credential-socket\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.647907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.647644 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-istio-token\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-istio-token\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.648960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.648709 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17604251-a04e-4687-8bd9-c164880f103a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17604251-a04e-4687-8bd9-c164880f103a-istiod-ca-cert\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.649193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.649120 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17604251-a04e-4687-8bd9-c164880f103a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17604251-a04e-4687-8bd9-c164880f103a-istio-podinfo\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.649428 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.649325 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-socket\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.649428 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.649374 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-data\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.651313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.649692 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-kube-api-access-pz9j9\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-kube-api-access-pz9j9\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.651313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.649769 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-certs\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.651313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.649831 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-envoy\") pod \"17604251-a04e-4687-8bd9-c164880f103a\" (UID: \"17604251-a04e-4687-8bd9-c164880f103a\") " Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652336 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652359 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652374 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652388 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652400 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652411 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0c98c9-ca4c-430f-a465-edc7b68da522-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652424 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqf6g\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-kube-api-access-vqf6g\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652436 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0c98c9-ca4c-430f-a465-edc7b68da522-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.654102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.652449 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0c98c9-ca4c-430f-a465-edc7b68da522-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.658524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.658492 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-credential-socket" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.660683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.660514 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-data" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.661129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.660880 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-socket" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.663567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.662054 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-certs" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.663787 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.663737 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-kube-api-access-pz9j9" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "kube-api-access-pz9j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.685772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.685736 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17604251-a04e-4687-8bd9-c164880f103a-istiod-ca-cert" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:46.689633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.689610 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-envoy" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.693344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.693309 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17604251-a04e-4687-8bd9-c164880f103a-istio-podinfo" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:46.696401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.696240 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-istio-token" pod "17604251-a04e-4687-8bd9-c164880f103a" (UID: "17604251-a04e-4687-8bd9-c164880f103a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.744455 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.744426 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:24:46.755116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755080 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17604251-a04e-4687-8bd9-c164880f103a-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.755116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755117 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.755311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755130 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.755311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755144 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9j9\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-kube-api-access-pz9j9\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.755311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755161 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.755311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755174 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.755311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755187 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17604251-a04e-4687-8bd9-c164880f103a-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.755311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755199 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17604251-a04e-4687-8bd9-c164880f103a-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.755311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.755215 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17604251-a04e-4687-8bd9-c164880f103a-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.856306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.856216 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-certs\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.856306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.856295 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-socket\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.856598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.856339 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-data\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.856598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.856406 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-credential-socket\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.856752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.856631 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-socket" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.856752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.856663 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-data" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.856875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.856855 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-podinfo\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.856960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.856929 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddccd18-fc91-4f95-8783-5042793d08ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddccd18-fc91-4f95-8783-5042793d08ac-istiod-ca-cert\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.857307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.857284 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-token\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-token\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.857506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.857491 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-envoy\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.857606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.857591 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-kube-api-access-xxvtz\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-kube-api-access-xxvtz\") pod \"0ddccd18-fc91-4f95-8783-5042793d08ac\" (UID: \"0ddccd18-fc91-4f95-8783-5042793d08ac\") " Oct 30 18:24:46.858587 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.858564 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.858672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.858589 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.862761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.862512 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-certs" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.862761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.862529 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-credential-socket" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.862897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.862869 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddccd18-fc91-4f95-8783-5042793d08ac-istiod-ca-cert" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:46.865141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.865105 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-podinfo" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:46.865459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.865428 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-envoy" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.865891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.865862 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-kube-api-access-xxvtz" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "kube-api-access-xxvtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.866201 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.866178 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-token" pod "0ddccd18-fc91-4f95-8783-5042793d08ac" (UID: "0ddccd18-fc91-4f95-8783-5042793d08ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.960209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.960098 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.960209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.960133 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.960209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.960151 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ddccd18-fc91-4f95-8783-5042793d08ac-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.960209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.960165 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.960209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.960173 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.960209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.960181 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxvtz\" (UniqueName: \"kubernetes.io/projected/0ddccd18-fc91-4f95-8783-5042793d08ac-kube-api-access-xxvtz\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.960209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.960189 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ddccd18-fc91-4f95-8783-5042793d08ac-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.965877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.965844 5624 generic.go:356] "Generic (PLEG): container finished" podID="17604251-a04e-4687-8bd9-c164880f103a" containerID="ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c" exitCode=0 Oct 30 18:24:46.966084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.965902 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" Oct 30 18:24:46.966084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.965929 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" event={"ID":"17604251-a04e-4687-8bd9-c164880f103a","Type":"ContainerDied","Data":"ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c"} Oct 30 18:24:46.966084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.966002 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g" event={"ID":"17604251-a04e-4687-8bd9-c164880f103a","Type":"ContainerDied","Data":"1f0e8c30d5622fbe240970dcc8d81242d64e45db042b48bacfc3f6998245eaf4"} Oct 30 18:24:46.966084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.966030 5624 scope.go:122] "RemoveContainer" containerID="ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c" Oct 30 18:24:46.968230 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.968204 5624 generic.go:356] "Generic (PLEG): container finished" podID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerID="850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273" exitCode=0 Oct 30 18:24:46.968461 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.968260 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" Oct 30 18:24:46.968461 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.968288 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" event={"ID":"1d0c98c9-ca4c-430f-a465-edc7b68da522","Type":"ContainerDied","Data":"850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273"} Oct 30 18:24:46.968461 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.968364 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65" event={"ID":"1d0c98c9-ca4c-430f-a465-edc7b68da522","Type":"ContainerDied","Data":"089b31fd5e5b60e81a8cf8f038eb0149911bb36b16a0f87092868294791fbb9d"} Oct 30 18:24:46.972059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.972035 5624 generic.go:356] "Generic (PLEG): container finished" podID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerID="78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608" exitCode=0 Oct 30 18:24:46.973018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.972079 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" Oct 30 18:24:46.973018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.972119 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" event={"ID":"0ddccd18-fc91-4f95-8783-5042793d08ac","Type":"ContainerDied","Data":"78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608"} Oct 30 18:24:46.973018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.972179 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz" event={"ID":"0ddccd18-fc91-4f95-8783-5042793d08ac","Type":"ContainerDied","Data":"094ab4d43b302cf7693e891c28aa866c2019fd016ce298757cf8efe63220eb5b"} Oct 30 18:24:46.977520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.975293 5624 generic.go:356] "Generic (PLEG): container finished" podID="fea5d15c-4957-49c0-9739-f5383e23f424" containerID="cc434f884a702dd2d77b7578742773d3dd3d5fe5c964b602f959a2cff97c725a" exitCode=0 Oct 30 18:24:46.977520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.975348 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" event={"ID":"fea5d15c-4957-49c0-9739-f5383e23f424","Type":"ContainerDied","Data":"cc434f884a702dd2d77b7578742773d3dd3d5fe5c964b602f959a2cff97c725a"} Oct 30 18:24:46.977520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.975859 5624 scope.go:122] "RemoveContainer" containerID="cc434f884a702dd2d77b7578742773d3dd3d5fe5c964b602f959a2cff97c725a" Oct 30 18:24:46.983370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.983346 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:24:46.983570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.983542 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:24:46.983673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.983609 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:24:46.984460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.983992 5624 scope.go:122] "RemoveContainer" containerID="ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c" Oct 30 18:24:46.984460 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:46.984345 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c\": container with ID starting with ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c not found: ID does not exist" containerID="ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c" Oct 30 18:24:46.984460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.984376 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c"} err="failed to get container status \"ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c\": rpc error: code = NotFound desc = could not find container \"ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c\": container with ID starting with ae307fcfa272e8f6287c808910d3e393a99a8b4ce0c2f38b9ef1939e2f22212c not found: ID does not exist" Oct 30 18:24:46.984460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:46.984396 5624 scope.go:122] "RemoveContainer" containerID="850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273" Oct 30 18:24:47.017748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.017723 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786"] Oct 30 18:24:47.026743 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.026724 5624 scope.go:122] "RemoveContainer" containerID="850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273" Oct 30 18:24:47.027117 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:47.027096 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273\": container with ID starting with 850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273 not found: ID does not exist" containerID="850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273" Oct 30 18:24:47.027270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.027247 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273"} err="failed to get container status \"850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273\": rpc error: code = NotFound desc = could not find container \"850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273\": container with ID starting with 850dbf516785fb200cc3b1d075f295398899da9c77812ec89f2b4162d30e7273 not found: ID does not exist" Oct 30 18:24:47.027397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.027273 5624 scope.go:122] "RemoveContainer" containerID="78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608" Oct 30 18:24:47.027397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.027152 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv"] Oct 30 18:24:47.039343 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.038174 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g"] Oct 30 18:24:47.052030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.052005 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-llf7g"] Oct 30 18:24:47.083467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.083444 5624 scope.go:122] "RemoveContainer" containerID="78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608" Oct 30 18:24:47.084020 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:47.083995 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608\": container with ID starting with 78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608 not found: ID does not exist" containerID="78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608" Oct 30 18:24:47.084098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.084040 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608"} err="failed to get container status \"78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608\": rpc error: code = NotFound desc = could not find container \"78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608\": container with ID starting with 78414842be288faa8a15b244052ee3e9db6d59ca729d16c3e5dd8401d2be5608 not found: ID does not exist" Oct 30 18:24:47.188963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.188696 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65"] Oct 30 18:24:47.199121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.199093 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-7nz65"] Oct 30 18:24:47.233242 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.233209 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz"] Oct 30 18:24:47.242449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.242415 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-d72zz"] Oct 30 18:24:47.280516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.280475 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:47.798620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.798575 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" path="/var/lib/kubelet/pods/0ddccd18-fc91-4f95-8783-5042793d08ac/volumes" Oct 30 18:24:47.799803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.799778 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17604251-a04e-4687-8bd9-c164880f103a" path="/var/lib/kubelet/pods/17604251-a04e-4687-8bd9-c164880f103a/volumes" Oct 30 18:24:47.800555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.800530 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" path="/var/lib/kubelet/pods/1d0c98c9-ca4c-430f-a465-edc7b68da522/volumes" Oct 30 18:24:47.985583 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.985546 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" event={"ID":"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb","Type":"ContainerStarted","Data":"162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1"} Oct 30 18:24:47.985769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.985602 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" event={"ID":"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb","Type":"ContainerStarted","Data":"d77d44071c6f24dbe6a214fca253f508b026925ecc38ea8c14e925dd5616d357"} Oct 30 18:24:47.985849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.985785 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:47.987722 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.987697 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" event={"ID":"4e202b77-f27a-4c86-84b0-ab8cee55c377","Type":"ContainerStarted","Data":"018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb"} Oct 30 18:24:47.987838 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.987741 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:47.987838 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.987766 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" event={"ID":"4e202b77-f27a-4c86-84b0-ab8cee55c377","Type":"ContainerStarted","Data":"eeab1290a8396e4b9868e91a1fd9e5c1ff72e748fea1e3605b2e7e028b6c22e5"} Oct 30 18:24:47.993531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:47.993512 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" event={"ID":"fea5d15c-4957-49c0-9739-f5383e23f424","Type":"ContainerStarted","Data":"1da3f66ff36bea22449d3ecd8175988bae0e3d9d8c90387d3ca0f55e744143fd"} Oct 30 18:24:48.028718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:48.028664 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" podStartSLOduration=3.028648667 podStartE2EDuration="3.028648667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:48.027591264 +0000 UTC m=+1382.416119382" watchObservedRunningTime="2026-10-30 18:24:48.028648667 +0000 UTC m=+1382.417176788" Oct 30 18:24:48.068861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:48.068764 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" podStartSLOduration=3.068748755 podStartE2EDuration="3.068748755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:48.067281949 +0000 UTC m=+1382.455810079" watchObservedRunningTime="2026-10-30 18:24:48.068748755 +0000 UTC m=+1382.457276875" Oct 30 18:24:48.622222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:48.622194 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:24:48.627279 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:48.627254 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:24:48.783423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:48.783393 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:24:48.784475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:48.784450 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:24:49.059596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:49.059526 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:49.723710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:49.723682 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:49.826272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:49.826240 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:49.917454 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:24:49.917422 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:24:49.953835 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:49.953809 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5"] Oct 30 18:24:50.184601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.184563 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:50.763465 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.763424 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:50.868141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.868106 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:50.932617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.932588 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp"] Oct 30 18:24:50.933517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933499 5624 cpu_manager.go:414] "containerMap: removing container" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" Oct 30 18:24:50.933517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933516 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" Oct 30 18:24:50.933638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933532 5624 cpu_manager.go:414] "containerMap: removing container" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" Oct 30 18:24:50.933638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933538 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" Oct 30 18:24:50.933638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933564 5624 cpu_manager.go:414] "containerMap: removing container" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" Oct 30 18:24:50.933638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933569 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" Oct 30 18:24:50.933764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933692 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0c98c9-ca4c-430f-a465-edc7b68da522" containerName="istio-proxy" Oct 30 18:24:50.933764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933708 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddccd18-fc91-4f95-8783-5042793d08ac" containerName="istio-proxy" Oct 30 18:24:50.933764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.933715 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="17604251-a04e-4687-8bd9-c164880f103a" containerName="istio-proxy" Oct 30 18:24:50.940457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.940439 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:50.957332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.957311 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp"] Oct 30 18:24:50.963105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.963080 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fb0abbf7\"" Oct 30 18:24:50.964626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:50.964605 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fb0abbf7-dockercfg-z67lg\"" Oct 30 18:24:51.115025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.099833 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6899f054-fc85-459e-a1a7-9dee926b2c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp\" (UID: \"6899f054-fc85-459e-a1a7-9dee926b2c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:51.115025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.099892 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2dg\" (UniqueName: \"kubernetes.io/projected/6899f054-fc85-459e-a1a7-9dee926b2c8c-kube-api-access-6h2dg\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp\" (UID: \"6899f054-fc85-459e-a1a7-9dee926b2c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:51.260350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.258462 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6899f054-fc85-459e-a1a7-9dee926b2c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp\" (UID: \"6899f054-fc85-459e-a1a7-9dee926b2c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:51.260350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.259281 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2dg\" (UniqueName: \"kubernetes.io/projected/6899f054-fc85-459e-a1a7-9dee926b2c8c-kube-api-access-6h2dg\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp\" (UID: \"6899f054-fc85-459e-a1a7-9dee926b2c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:51.268956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.262778 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6899f054-fc85-459e-a1a7-9dee926b2c8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp\" (UID: \"6899f054-fc85-459e-a1a7-9dee926b2c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:51.278086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.278048 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm"] Oct 30 18:24:51.322893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.322861 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm"] Oct 30 18:24:51.336980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.328229 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:51.342994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.340303 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2dg\" (UniqueName: \"kubernetes.io/projected/6899f054-fc85-459e-a1a7-9dee926b2c8c-kube-api-access-6h2dg\") pod \"payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp\" (UID: \"6899f054-fc85-459e-a1a7-9dee926b2c8c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:51.369604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.369526 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67687132-9d0b-4ebe-b344-3881f9369509-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm\" (UID: \"67687132-9d0b-4ebe-b344-3881f9369509\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:51.369744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.369679 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kczfp\" (UniqueName: \"kubernetes.io/projected/67687132-9d0b-4ebe-b344-3881f9369509-kube-api-access-kczfp\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm\" (UID: \"67687132-9d0b-4ebe-b344-3881f9369509\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:51.422025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.421982 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:51.471238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.471199 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kczfp\" (UniqueName: \"kubernetes.io/projected/67687132-9d0b-4ebe-b344-3881f9369509-kube-api-access-kczfp\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm\" (UID: \"67687132-9d0b-4ebe-b344-3881f9369509\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:51.471577 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.471558 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67687132-9d0b-4ebe-b344-3881f9369509-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm\" (UID: \"67687132-9d0b-4ebe-b344-3881f9369509\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:51.472517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.472490 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67687132-9d0b-4ebe-b344-3881f9369509-plugins-config-volume\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm\" (UID: \"67687132-9d0b-4ebe-b344-3881f9369509\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:51.494440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.494347 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kczfp\" (UniqueName: \"kubernetes.io/projected/67687132-9d0b-4ebe-b344-3881f9369509-kube-api-access-kczfp\") pod \"payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm\" (UID: \"67687132-9d0b-4ebe-b344-3881f9369509\") " pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:51.558803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.558777 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:51.699531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.699503 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:51.897652 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:51.897594 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6899f054_fc85_459e_a1a7_9dee926b2c8c.slice/crio-cf3cc26ef291db7fae083ad7cf26b5397b6a2b126b50778471ad2aa00a438f54 WatchSource:0}: Error finding container cf3cc26ef291db7fae083ad7cf26b5397b6a2b126b50778471ad2aa00a438f54: Status 404 returned error can't find the container with id cf3cc26ef291db7fae083ad7cf26b5397b6a2b126b50778471ad2aa00a438f54 Oct 30 18:24:51.901420 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:51.901348 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp"] Oct 30 18:24:52.032049 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:52.032012 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67687132_9d0b_4ebe_b344_3881f9369509.slice/crio-29db87bb6a7ecbb07fdbe296a7a464ba54080446a50674691f5574c6e2eafe30 WatchSource:0}: Error finding container 29db87bb6a7ecbb07fdbe296a7a464ba54080446a50674691f5574c6e2eafe30: Status 404 returned error can't find the container with id 29db87bb6a7ecbb07fdbe296a7a464ba54080446a50674691f5574c6e2eafe30 Oct 30 18:24:52.035573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.035550 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm"] Oct 30 18:24:52.188444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.188370 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" event={"ID":"67687132-9d0b-4ebe-b344-3881f9369509","Type":"ContainerStarted","Data":"87a04391c907d3c7f82437215de10c9acd1c364244db569696ac56c1f753deb8"} Oct 30 18:24:52.188444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.188434 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" event={"ID":"67687132-9d0b-4ebe-b344-3881f9369509","Type":"ContainerStarted","Data":"29db87bb6a7ecbb07fdbe296a7a464ba54080446a50674691f5574c6e2eafe30"} Oct 30 18:24:52.188635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.188461 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:24:52.190193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.190155 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" event={"ID":"6899f054-fc85-459e-a1a7-9dee926b2c8c","Type":"ContainerStarted","Data":"f58950d044941ac00904e7ab1452f614f36b98bfd51ff4d68f4ca1cb3bc69180"} Oct 30 18:24:52.190311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.190211 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" event={"ID":"6899f054-fc85-459e-a1a7-9dee926b2c8c","Type":"ContainerStarted","Data":"cf3cc26ef291db7fae083ad7cf26b5397b6a2b126b50778471ad2aa00a438f54"} Oct 30 18:24:52.190311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.190243 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:24:52.235336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.235290 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" podStartSLOduration=1.23527831 podStartE2EDuration="1.23527831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:52.229366066 +0000 UTC m=+1386.617894183" watchObservedRunningTime="2026-10-30 18:24:52.23527831 +0000 UTC m=+1386.623806426" Oct 30 18:24:52.282304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:52.282257 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" podStartSLOduration=2.2822448 podStartE2EDuration="2.2822448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:52.275903017 +0000 UTC m=+1386.664431135" watchObservedRunningTime="2026-10-30 18:24:52.2822448 +0000 UTC m=+1386.670772944" Oct 30 18:24:53.527732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:53.527704 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:54.667631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:54.667595 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:54.709794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:54.709762 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:24:55.316061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.316028 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:55.421289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.421174 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:55.768397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.768366 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p"] Oct 30 18:24:55.776178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.775433 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.804565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.804533 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p"] Oct 30 18:24:55.805902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.805878 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-227c01-openshift-default-dockercfg-n8qwb\"" Oct 30 18:24:55.827712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.825642 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:55.874384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.874354 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istiod-ca-cert\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.874530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.874408 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-data\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.874575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.874537 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-socket\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.874612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.874589 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-certs\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.874656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.874636 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-envoy\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.874744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.874727 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gck2m\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-kube-api-access-gck2m\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.874842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.874827 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-token\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.875006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.874992 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-credential-socket\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.875551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.875039 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-podinfo\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:55.941092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.941056 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:55.977272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.977216 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-credential-socket\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.003014 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-podinfo\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.003242 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istiod-ca-cert\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.003315 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-data\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.003403 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-socket\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.003452 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-certs\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.003476 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-envoy\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.003558 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gck2m\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-kube-api-access-gck2m\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.003592 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-token\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.004957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:55.998414 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-credential-socket\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.005923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.005898 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-socket\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.006678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.006658 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istiod-ca-cert\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.007046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.007028 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-data\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.021024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.009525 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-certs\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.021246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.013837 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-podinfo\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.028184 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.025475 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-mndc5" Oct 30 18:24:56.033409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.033198 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-envoy\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.052230 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.048458 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gck2m\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-kube-api-access-gck2m\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.061966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.054387 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-token\") pod \"e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.112968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.109093 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:56.370008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.366038 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:56.483093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.481019 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:56.693014 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.692984 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr"] Oct 30 18:24:56.693287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.693245 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" containerID="cri-o://d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4" gracePeriod=30 Oct 30 18:24:56.706754 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:24:56.706722 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3d9000_f1f2_450b_9da7_c1de20b34b77.slice/crio-944806e00af476008849a8f2c0856748868f207a2dce0c6e353815afd51bde57 WatchSource:0}: Error finding container 944806e00af476008849a8f2c0856748868f207a2dce0c6e353815afd51bde57: Status 404 returned error can't find the container with id 944806e00af476008849a8f2c0856748868f207a2dce0c6e353815afd51bde57 Oct 30 18:24:56.707202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.707178 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p"] Oct 30 18:24:56.710287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.710244 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:24:56.710448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.710328 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:24:56.710448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:56.710371 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:24:57.426387 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:57.426353 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" event={"ID":"9e3d9000-f1f2-450b-9da7-c1de20b34b77","Type":"ContainerStarted","Data":"0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f"} Oct 30 18:24:57.427369 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:57.426417 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" event={"ID":"9e3d9000-f1f2-450b-9da7-c1de20b34b77","Type":"ContainerStarted","Data":"944806e00af476008849a8f2c0856748868f207a2dce0c6e353815afd51bde57"} Oct 30 18:24:57.477255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:57.477210 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podStartSLOduration=2.477197811 podStartE2EDuration="2.477197811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:57.472409045 +0000 UTC m=+1391.860937159" watchObservedRunningTime="2026-10-30 18:24:57.477197811 +0000 UTC m=+1391.865725927" Oct 30 18:24:57.697833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:57.697400 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:58.122596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:58.122568 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:58.129795 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:58.129774 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:58.435891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:58.435821 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:58.436769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:58.436748 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:24:59.004914 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:59.004879 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:24:59.005288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:59.005062 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:24:59.567681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:24:59.567635 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:00.009105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:00.008989 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:00.133876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:00.133814 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:01.523577 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:01.523544 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:01.747080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:01.747052 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:01.840873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:01.840840 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:01.919676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:01.919643 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4"] Oct 30 18:25:01.924869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:01.924847 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:01.958117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:01.957222 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-227c01-dockercfg-sd9cl\"" Oct 30 18:25:01.958117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:01.957465 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-227c01\"" Oct 30 18:25:01.958117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:01.958006 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4"] Oct 30 18:25:02.131058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.083686 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7c8\" (UniqueName: \"kubernetes.io/projected/2c427900-f342-4cff-878b-4624a9165ae5-kube-api-access-lg7c8\") pod \"payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4\" (UID: \"2c427900-f342-4cff-878b-4624a9165ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:02.131058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.083856 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c427900-f342-4cff-878b-4624a9165ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4\" (UID: \"2c427900-f342-4cff-878b-4624a9165ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:02.203003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.198606 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7c8\" (UniqueName: \"kubernetes.io/projected/2c427900-f342-4cff-878b-4624a9165ae5-kube-api-access-lg7c8\") pod \"payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4\" (UID: \"2c427900-f342-4cff-878b-4624a9165ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:02.203003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.198769 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c427900-f342-4cff-878b-4624a9165ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4\" (UID: \"2c427900-f342-4cff-878b-4624a9165ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:02.203352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.203335 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb"] Oct 30 18:25:02.214983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.210048 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c427900-f342-4cff-878b-4624a9165ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4\" (UID: \"2c427900-f342-4cff-878b-4624a9165ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:02.389695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.386279 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:02.392889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.388337 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7c8\" (UniqueName: \"kubernetes.io/projected/2c427900-f342-4cff-878b-4624a9165ae5-kube-api-access-lg7c8\") pod \"payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4\" (UID: \"2c427900-f342-4cff-878b-4624a9165ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:02.402342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.402310 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb"] Oct 30 18:25:02.452780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.452757 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:25:02.474060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.474031 5624 generic.go:356] "Generic (PLEG): container finished" podID="023e0604-0666-4759-a030-cf93f46ebb78" containerID="d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4" exitCode=0 Oct 30 18:25:02.474197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.474072 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" event={"ID":"023e0604-0666-4759-a030-cf93f46ebb78","Type":"ContainerDied","Data":"d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4"} Oct 30 18:25:02.474197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.474126 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" event={"ID":"023e0604-0666-4759-a030-cf93f46ebb78","Type":"ContainerDied","Data":"1927baeed3fa91cf612ebede25614778dad743d8e25518e4cb31d9eefc2e573c"} Oct 30 18:25:02.474197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.474151 5624 scope.go:122] "RemoveContainer" containerID="d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4" Oct 30 18:25:02.474315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.474083 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr" Oct 30 18:25:02.476165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.476141 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgdp\" (UniqueName: \"kubernetes.io/projected/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-kube-api-access-wmgdp\") pod \"payload-processing-e2e-api-b-227c01-cb946789d-9nbjb\" (UID: \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:02.476395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.476379 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-227c01-cb946789d-9nbjb\" (UID: \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:02.486761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.486742 5624 scope.go:122] "RemoveContainer" containerID="d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4" Oct 30 18:25:02.487085 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:02.487064 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4\": container with ID starting with d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4 not found: ID does not exist" containerID="d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4" Oct 30 18:25:02.487141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.487093 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4"} err="failed to get container status \"d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4\": rpc error: code = NotFound desc = could not find container \"d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4\": container with ID starting with d516f227f13106e1cb545f6de28072a8898ce5657b500a010577ecbb7741e0e4 not found: ID does not exist" Oct 30 18:25:02.577213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577183 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-certs\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.577790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577296 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-credential-socket\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.577790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577330 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023e0604-0666-4759-a030-cf93f46ebb78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023e0604-0666-4759-a030-cf93f46ebb78-istiod-ca-cert\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.577790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577412 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-kube-api-access-6dkj4\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-kube-api-access-6dkj4\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.577790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577492 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-istio-token\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-istio-token\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.577790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577511 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-data\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.577790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577610 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-socket\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.577790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577652 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/023e0604-0666-4759-a030-cf93f46ebb78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023e0604-0666-4759-a030-cf93f46ebb78-istio-podinfo\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.577790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.577735 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-envoy\") pod \"023e0604-0666-4759-a030-cf93f46ebb78\" (UID: \"023e0604-0666-4759-a030-cf93f46ebb78\") " Oct 30 18:25:02.578912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.578124 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgdp\" (UniqueName: \"kubernetes.io/projected/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-kube-api-access-wmgdp\") pod \"payload-processing-e2e-api-b-227c01-cb946789d-9nbjb\" (UID: \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:02.578912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.578279 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-227c01-cb946789d-9nbjb\" (UID: \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:02.579817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.579791 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-227c01-cb946789d-9nbjb\" (UID: \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:02.580926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.580894 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-socket" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.580926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.580927 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-certs" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.581383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.580911 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-data" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.581508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.581393 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-credential-socket" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.581621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.581530 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023e0604-0666-4759-a030-cf93f46ebb78-istiod-ca-cert" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:02.583371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.583345 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-istio-token" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.585508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.585034 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-envoy" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.585508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.585117 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/023e0604-0666-4759-a030-cf93f46ebb78-istio-podinfo" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:02.590884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.590859 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-kube-api-access-6dkj4" pod "023e0604-0666-4759-a030-cf93f46ebb78" (UID: "023e0604-0666-4759-a030-cf93f46ebb78"). InnerVolumeSpecName "kube-api-access-6dkj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.601018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.600996 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgdp\" (UniqueName: \"kubernetes.io/projected/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-kube-api-access-wmgdp\") pod \"payload-processing-e2e-api-b-227c01-cb946789d-9nbjb\" (UID: \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:02.662625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.662543 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:02.680256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680229 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.680256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680253 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023e0604-0666-4759-a030-cf93f46ebb78-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.680256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680265 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.680256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680273 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.680586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680281 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.680586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680291 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023e0604-0666-4759-a030-cf93f46ebb78-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.680586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680299 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dkj4\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-kube-api-access-6dkj4\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.680586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680307 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023e0604-0666-4759-a030-cf93f46ebb78-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.680586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.680315 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023e0604-0666-4759-a030-cf93f46ebb78-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.752005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.751974 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:02.918470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.918356 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr"] Oct 30 18:25:02.925721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:02.925699 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nc6kr"] Oct 30 18:25:03.019984 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:25:03.019922 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c427900_f342_4cff_878b_4624a9165ae5.slice/crio-7dd7be866e7ecd80a5eac9df4acb81d6626392fe3fed6cf540c8a0f3c305659c WatchSource:0}: Error finding container 7dd7be866e7ecd80a5eac9df4acb81d6626392fe3fed6cf540c8a0f3c305659c: Status 404 returned error can't find the container with id 7dd7be866e7ecd80a5eac9df4acb81d6626392fe3fed6cf540c8a0f3c305659c Oct 30 18:25:03.022318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.022293 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4"] Oct 30 18:25:03.135650 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:25:03.135608 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ea3dea_cf99_46ba_8ae0_7e827da688e8.slice/crio-8e6a67778f022672ee77a5f385460a92be8de57993425523a03b37e7e445941e WatchSource:0}: Error finding container 8e6a67778f022672ee77a5f385460a92be8de57993425523a03b37e7e445941e: Status 404 returned error can't find the container with id 8e6a67778f022672ee77a5f385460a92be8de57993425523a03b37e7e445941e Oct 30 18:25:03.138462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.138438 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb"] Oct 30 18:25:03.260164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.260136 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:25:03.260334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.260224 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:25:03.640751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.636113 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" event={"ID":"b1ea3dea-cf99-46ba-8ae0-7e827da688e8","Type":"ContainerStarted","Data":"28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360"} Oct 30 18:25:03.640751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.636200 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" event={"ID":"b1ea3dea-cf99-46ba-8ae0-7e827da688e8","Type":"ContainerStarted","Data":"8e6a67778f022672ee77a5f385460a92be8de57993425523a03b37e7e445941e"} Oct 30 18:25:03.640751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.637774 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:03.739986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.738419 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" event={"ID":"2c427900-f342-4cff-878b-4624a9165ae5","Type":"ContainerStarted","Data":"057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72"} Oct 30 18:25:03.739986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.738488 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" event={"ID":"2c427900-f342-4cff-878b-4624a9165ae5","Type":"ContainerStarted","Data":"7dd7be866e7ecd80a5eac9df4acb81d6626392fe3fed6cf540c8a0f3c305659c"} Oct 30 18:25:03.739986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.739752 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:03.749961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.746717 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:03.815430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.804023 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" podStartSLOduration=2.804000372 podStartE2EDuration="2.804000372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:03.695752321 +0000 UTC m=+1398.084280441" watchObservedRunningTime="2026-10-30 18:25:03.804000372 +0000 UTC m=+1398.192528492" Oct 30 18:25:03.815430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.808762 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" podStartSLOduration=2.80874349 podStartE2EDuration="2.80874349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:03.80716052 +0000 UTC m=+1398.195688639" watchObservedRunningTime="2026-10-30 18:25:03.80874349 +0000 UTC m=+1398.197271609" Oct 30 18:25:03.864924 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:03.860877 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023e0604-0666-4759-a030-cf93f46ebb78" path="/var/lib/kubelet/pods/023e0604-0666-4759-a030-cf93f46ebb78/volumes" Oct 30 18:25:04.997694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:04.997656 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:04.998146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:04.998044 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" containerID="cri-o://5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09" gracePeriod=45 Oct 30 18:25:05.349360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.349336 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:25:05.374041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.374012 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c9af44-a4f5-4014-aac6-9116897541e0-kube-api-access-llkv5\" (UniqueName: \"kubernetes.io/projected/16c9af44-a4f5-4014-aac6-9116897541e0-kube-api-access-llkv5\") pod \"16c9af44-a4f5-4014-aac6-9116897541e0\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " Oct 30 18:25:05.374216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.374096 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-home\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-home\") pod \"16c9af44-a4f5-4014-aac6-9116897541e0\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " Oct 30 18:25:05.374216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.374166 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-dshm\") pod \"16c9af44-a4f5-4014-aac6-9116897541e0\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " Oct 30 18:25:05.374216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.374195 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-model-cache\") pod \"16c9af44-a4f5-4014-aac6-9116897541e0\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " Oct 30 18:25:05.374402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.374304 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-tmp-dir\") pod \"16c9af44-a4f5-4014-aac6-9116897541e0\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " Oct 30 18:25:05.374402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.374337 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c9af44-a4f5-4014-aac6-9116897541e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/16c9af44-a4f5-4014-aac6-9116897541e0-tls-certs\") pod \"16c9af44-a4f5-4014-aac6-9116897541e0\" (UID: \"16c9af44-a4f5-4014-aac6-9116897541e0\") " Oct 30 18:25:05.374884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.374780 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-model-cache" pod "16c9af44-a4f5-4014-aac6-9116897541e0" (UID: "16c9af44-a4f5-4014-aac6-9116897541e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.375550 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.375253 5624 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-model-cache\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.375550 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.375514 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-home" pod "16c9af44-a4f5-4014-aac6-9116897541e0" (UID: "16c9af44-a4f5-4014-aac6-9116897541e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.375814 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.375778 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-tmp-dir" pod "16c9af44-a4f5-4014-aac6-9116897541e0" (UID: "16c9af44-a4f5-4014-aac6-9116897541e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.376995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.376929 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c9af44-a4f5-4014-aac6-9116897541e0-kube-api-access-llkv5" pod "16c9af44-a4f5-4014-aac6-9116897541e0" (UID: "16c9af44-a4f5-4014-aac6-9116897541e0"). InnerVolumeSpecName "kube-api-access-llkv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:05.377138 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.377054 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-dshm" pod "16c9af44-a4f5-4014-aac6-9116897541e0" (UID: "16c9af44-a4f5-4014-aac6-9116897541e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:05.378379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.378355 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c9af44-a4f5-4014-aac6-9116897541e0-tls-certs" pod "16c9af44-a4f5-4014-aac6-9116897541e0" (UID: "16c9af44-a4f5-4014-aac6-9116897541e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:05.477015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.476979 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llkv5\" (UniqueName: \"kubernetes.io/projected/16c9af44-a4f5-4014-aac6-9116897541e0-kube-api-access-llkv5\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.477015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.477016 5624 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-home\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.478052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.477034 5624 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-dshm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.478052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.477047 5624 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c9af44-a4f5-4014-aac6-9116897541e0-tmp-dir\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.478052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.477059 5624 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c9af44-a4f5-4014-aac6-9116897541e0-tls-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:05.490314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.490284 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm"] Oct 30 18:25:05.491634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.491608 5624 cpu_manager.go:414] "containerMap: removing container" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" Oct 30 18:25:05.491737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.491634 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" Oct 30 18:25:05.491737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.491725 5624 cpu_manager.go:414] "containerMap: removing container" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" Oct 30 18:25:05.491860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.491735 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" Oct 30 18:25:05.492012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.491992 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="023e0604-0666-4759-a030-cf93f46ebb78" containerName="istio-proxy" Oct 30 18:25:05.492081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.492030 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" Oct 30 18:25:05.496899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.496874 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.517999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.517696 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-550276-serving-cert\"" Oct 30 18:25:05.527711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.527682 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm"] Oct 30 18:25:05.575309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.575225 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:05.578081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.578060 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbh9r\" (UniqueName: \"kubernetes.io/projected/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-kube-api-access-qbh9r\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.578206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.578177 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-metrics-certs\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.578252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.578235 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-openshift-service-ca\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.578303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.578264 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-maas-api-tls\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.674920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.674893 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:05.679573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.679545 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-openshift-service-ca\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.679713 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.679587 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-maas-api-tls\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.679810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.679793 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbh9r\" (UniqueName: \"kubernetes.io/projected/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-kube-api-access-qbh9r\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.679896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.679881 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-metrics-certs\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.680233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.680211 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-openshift-service-ca\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.682125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.682103 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-maas-api-tls\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.682501 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.682257 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-metrics-certs\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.703803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.703781 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbh9r\" (UniqueName: \"kubernetes.io/projected/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-kube-api-access-qbh9r\") pod \"maas-api-e2e-api-b-550276-678dff9d59-9snzm\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.757531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.757500 5624 generic.go:356] "Generic (PLEG): container finished" podID="16c9af44-a4f5-4014-aac6-9116897541e0" containerID="5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09" exitCode=0 Oct 30 18:25:05.757712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.757549 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" event={"ID":"16c9af44-a4f5-4014-aac6-9116897541e0","Type":"ContainerDied","Data":"5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09"} Oct 30 18:25:05.757803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.757574 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" Oct 30 18:25:05.757975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.757921 5624 scope.go:122] "RemoveContainer" containerID="5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09" Oct 30 18:25:05.772558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.758788 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" event={"ID":"16c9af44-a4f5-4014-aac6-9116897541e0","Type":"ContainerDied","Data":"e938d2993f1fd992fc005dcfe76f929048532c3985e827cba55138214c80b423"} Oct 30 18:25:05.797599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.797546 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:05.835472 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.835202 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:05.867432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.867403 5624 scope.go:122] "RemoveContainer" containerID="5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09" Oct 30 18:25:05.867836 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:05.867811 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09\": container with ID starting with 5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09 not found: ID does not exist" containerID="5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09" Oct 30 18:25:05.867996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.867843 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09"} err="failed to get container status \"5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09\": rpc error: code = NotFound desc = could not find container \"5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09\": container with ID starting with 5d5090738a3142d0aca8b326595871402ec7beadd475cb991928a7b19cfedc09 not found: ID does not exist" Oct 30 18:25:05.913599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.913515 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5"] Oct 30 18:25:05.958840 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:05.958798 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5"] Oct 30 18:25:06.227901 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:25:06.227869 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a22e04_5ee6_4bd4_8848_6f8f96a65eac.slice/crio-d7add461e1c69a07cb60eca4250651f871071d782fece8eaab27b405337579a3 WatchSource:0}: Error finding container d7add461e1c69a07cb60eca4250651f871071d782fece8eaab27b405337579a3: Status 404 returned error can't find the container with id d7add461e1c69a07cb60eca4250651f871071d782fece8eaab27b405337579a3 Oct 30 18:25:06.229978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:06.229951 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm"] Oct 30 18:25:06.299959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:06.299910 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-fdf4da/distinct-w5-fdf4da-kserve-8577fbc559-52sr5" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 30 18:25:06.905064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:06.905031 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" event={"ID":"41a22e04-5ee6-4bd4-8848-6f8f96a65eac","Type":"ContainerStarted","Data":"d7add461e1c69a07cb60eca4250651f871071d782fece8eaab27b405337579a3"} Oct 30 18:25:07.640259 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:07.640226 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:07.830660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:07.830626 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c9af44-a4f5-4014-aac6-9116897541e0" path="/var/lib/kubelet/pods/16c9af44-a4f5-4014-aac6-9116897541e0/volumes" Oct 30 18:25:07.917822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:07.917739 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" event={"ID":"41a22e04-5ee6-4bd4-8848-6f8f96a65eac","Type":"ContainerStarted","Data":"49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6"} Oct 30 18:25:07.918019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:07.917862 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:07.964274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:07.964222 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" podStartSLOduration=2.61098222 podStartE2EDuration="2.96420931s" totalImagesPullingTime="353.22709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:07.960621623 +0000 UTC m=+1402.349149743" watchObservedRunningTime="2026-10-30 18:25:07.96420931 +0000 UTC m=+1402.352737445" Oct 30 18:25:09.404187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:09.404153 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:10.932475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:10.932439 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d"] Oct 30 18:25:10.938085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:10.938053 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:10.976517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:10.970008 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fb0abbf7-serving-cert\"" Oct 30 18:25:10.979285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:10.979254 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d"] Oct 30 18:25:11.021112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.021046 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:11.047266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.047235 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-metrics-certs\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.047433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.047410 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8cc09-b308-4348-af04-5773d5134665-openshift-service-ca\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.047582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.047566 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.047667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.047616 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txvn\" (UniqueName: \"kubernetes.io/projected/ffa8cc09-b308-4348-af04-5773d5134665-kube-api-access-5txvn\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.144466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.144301 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:11.159206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.149809 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.159206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.149885 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txvn\" (UniqueName: \"kubernetes.io/projected/ffa8cc09-b308-4348-af04-5773d5134665-kube-api-access-5txvn\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.161638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.159599 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-metrics-certs\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.161638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.159895 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8cc09-b308-4348-af04-5773d5134665-openshift-service-ca\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.161638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.160743 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8cc09-b308-4348-af04-5773d5134665-openshift-service-ca\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.161638 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:11.161337 5624 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-serving-cert: secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:25:11.161638 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:11.161404 5624 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls podName:ffa8cc09-b308-4348-af04-5773d5134665 nodeName:}" failed. No retries permitted until 2026-10-30 18:25:11.661383659 +0000 UTC m=+1406.049911776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls") pod "maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" (UID: "ffa8cc09-b308-4348-af04-5773d5134665") : secret "maas-api-e2e-mt-fb0abbf7-serving-cert" not found Oct 30 18:25:11.164792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.164431 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-metrics-certs\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.182513 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.182440 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txvn\" (UniqueName: \"kubernetes.io/projected/ffa8cc09-b308-4348-af04-5773d5134665-kube-api-access-5txvn\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.340097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.340060 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:11.673608 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.673575 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.676177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.676151 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls\") pod \"maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:11.857503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:11.857459 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:12.222399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:12.222308 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d"] Oct 30 18:25:12.832949 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:12.832856 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:12.989692 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:12.989647 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" event={"ID":"ffa8cc09-b308-4348-af04-5773d5134665","Type":"ContainerStarted","Data":"97dc996afcb9b67f5197e84052ed231f0d35daea3941f3f575bab27bc5d36b4f"} Oct 30 18:25:12.990010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:12.989984 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" event={"ID":"ffa8cc09-b308-4348-af04-5773d5134665","Type":"ContainerStarted","Data":"a56e5eb741bf733477201e9dec4f3f408ce1c8af533f2705c338da4ee89eb716"} Oct 30 18:25:12.991453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:12.991426 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:13.039471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:13.039424 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" podStartSLOduration=2.643458803 podStartE2EDuration="3.039409521s" totalImagesPullingTime="395.950718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:13.038304743 +0000 UTC m=+1407.426832861" watchObservedRunningTime="2026-10-30 18:25:13.039409521 +0000 UTC m=+1407.427937690" Oct 30 18:25:13.933832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:13.933805 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:25:14.433297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:14.433260 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:15.761661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:15.761626 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:25:15.762186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:15.762104 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:25:15.890134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:15.890100 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:25:16.038984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:16.038881 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:17.325845 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:17.325797 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:17.426366 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:17.426331 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:17.913958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:17.913547 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7"] Oct 30 18:25:17.933543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:17.933515 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:17.941065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:17.940288 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7"] Oct 30 18:25:18.112148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.103404 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1470d092-0818-4480-a80a-00b91adbeb4f-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.140542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.140167 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.140852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.140832 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.141170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.141149 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.167223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.167093 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1470d092-0818-4480-a80a-00b91adbeb4f-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.479013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.434156 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.479013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.434271 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.479013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.434495 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.479013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.434551 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jqhn\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-kube-api-access-2jqhn\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.732798 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.732949 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.732986 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1470d092-0818-4480-a80a-00b91adbeb4f-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.733023 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.733080 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.733242 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.733284 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqhn\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-kube-api-access-2jqhn\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.733410 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1470d092-0818-4480-a80a-00b91adbeb4f-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.733532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.733460 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.782411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.733903 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.782411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.734247 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.782411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.734480 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.782411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.749980 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.782411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.750895 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1470d092-0818-4480-a80a-00b91adbeb4f-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.814023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.800622 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1470d092-0818-4480-a80a-00b91adbeb4f-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.861183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.854282 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.861183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.854771 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:18.865739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:18.864008 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqhn\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-kube-api-access-2jqhn\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:19.032347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.032265 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:19.241830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.241796 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:19.411967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.402379 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:19.461298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.457634 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7"] Oct 30 18:25:19.466477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.466429 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:25:19.466611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.466533 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:25:19.466611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.466583 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:25:19.650808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.650772 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" event={"ID":"1470d092-0818-4480-a80a-00b91adbeb4f","Type":"ContainerStarted","Data":"6ee95902cb4761d34ba6b0dff76a995fe0a7b2d1f8cccb3753de6e51d22eb142"} Oct 30 18:25:19.843342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:19.843230 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:19.961042 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:19.961001 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:25:20.038594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:20.038557 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:25:20.467600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:20.467502 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:20.599917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:20.598824 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:20.783042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:20.782142 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" event={"ID":"1470d092-0818-4480-a80a-00b91adbeb4f","Type":"ContainerStarted","Data":"3d3a30f983dc3c6ce28482eea1d1290d6365c230dead46a3e9197a3f02d77f32"} Oct 30 18:25:20.924960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:20.903876 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" podStartSLOduration=3.903854823 podStartE2EDuration="3.903854823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:20.865444372 +0000 UTC m=+1415.253972492" watchObservedRunningTime="2026-10-30 18:25:20.903854823 +0000 UTC m=+1415.292382943" Oct 30 18:25:21.043117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:21.039374 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:21.043117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:21.039537 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:25:21.043117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:21.039582 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 18:25:21.947425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:21.945615 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:22.040959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:22.040092 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:25:22.040959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:22.040148 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 18:25:23.039913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:23.039878 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:25:23.040518 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:23.040490 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 18:25:24.006355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:24.006322 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:24.036348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:24.036310 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:25:24.036664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:24.036633 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 18:25:25.032977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:25.032861 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:25:25.032977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:25.032955 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 18:25:26.032775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:26.032733 5624 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 18:25:26.032987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:26.032822 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 18:25:26.460590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:26.460552 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:27.040961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:27.040865 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:27.051164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:27.045884 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:27.051164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:27.050652 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:27.051842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:27.051817 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:25:27.172331 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:27.172296 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:28.027694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:28.027656 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:28.123139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:28.123106 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:28.734544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:28.716067 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:29.273799 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:29.273766 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:29.386521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:29.386493 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:29.936130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:29.936076 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:30.959954 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:30.959907 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:31.070707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:31.070629 5624 generic.go:356] "Generic (PLEG): container finished" podID="b51cc775-0d90-40a5-9c60-5d802467532a" containerID="1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e" exitCode=0 Oct 30 18:25:31.070860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:31.070700 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" event={"ID":"b51cc775-0d90-40a5-9c60-5d802467532a","Type":"ContainerDied","Data":"1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e"} Oct 30 18:25:31.071138 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:31.071123 5624 scope.go:122] "RemoveContainer" containerID="1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e" Oct 30 18:25:31.072881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:31.072854 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:25:31.072966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:31.072926 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:25:31.073012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:31.072992 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:25:32.081969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:32.081919 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" event={"ID":"b51cc775-0d90-40a5-9c60-5d802467532a","Type":"ContainerStarted","Data":"7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e"} Oct 30 18:25:32.130476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:32.130443 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:25:32.130656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:32.130632 5624 patch_prober.go:29] interesting pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:32.130701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:32.130678 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 18:25:32.781919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:32.781884 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:33.130757 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:33.130378 5624 patch_prober.go:29] interesting pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:33.130757 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:33.130436 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 18:25:34.130202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:34.130166 5624 patch_prober.go:29] interesting pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:34.130424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:34.130226 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 18:25:34.533065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:34.530372 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:35.145955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:35.145895 5624 patch_prober.go:29] interesting pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:35.146180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:35.145978 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 18:25:35.788957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:35.785751 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:36.013682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:36.010842 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:36.112476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:36.112367 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:36.130775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:36.130311 5624 patch_prober.go:29] interesting pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:36.130775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:36.130389 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 18:25:36.320188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:36.320135 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:36.615844 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:36.615538 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:37.130570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:37.130533 5624 patch_prober.go:29] interesting pod/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 18:25:37.131110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:37.130606 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 18:25:38.049896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:38.049861 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:25:38.135057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:38.135026 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:25:38.362698 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:38.362624 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:25:39.111178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:39.111142 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:39.207123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:39.207094 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:39.333955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:39.327656 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:41.661981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:41.661925 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:42.081794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:42.081715 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.254072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.254043 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.360918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.360881 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.459812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.459784 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.620484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.617797 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:43.669768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.669731 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx"] Oct 30 18:25:43.670036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.670008 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" containerID="cri-o://897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d" gracePeriod=30 Oct 30 18:25:43.679198 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.679174 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr"] Oct 30 18:25:43.679434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.679385 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" containerID="cri-o://255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c" gracePeriod=30 Oct 30 18:25:43.711691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.711289 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d"] Oct 30 18:25:43.711691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:43.711590 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" containerID="cri-o://29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada" gracePeriod=30 Oct 30 18:25:45.123136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:45.123096 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6"] Oct 30 18:25:45.146979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:45.146952 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:45.158164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:45.158139 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd"] Oct 30 18:25:45.345765 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:45.345732 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:45.446121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:45.446040 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:46.842273 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:46.842239 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:46.842916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:46.842875 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:47.421957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:47.420845 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:47.421957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:47.420904 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:47.834757 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:47.834664 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:47.974959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:47.964999 5624 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:25:47.974959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:47.965072 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:25:49.197958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.194600 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:25:49.261958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.261860 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-credential-socket\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.262115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.261980 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-podinfo\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.262168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262153 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-kube-api-access-8zx9q\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-kube-api-access-8zx9q\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262262 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-certs\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262341 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-data\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262411 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4d072e-56c8-4f49-966c-18445c91eea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4d072e-56c8-4f49-966c-18445c91eea2-istiod-ca-cert\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262455 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-socket\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262519 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-credential-socket" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262639 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-certs" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262663 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-envoy\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262693 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-token\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-token\") pod \"cc4d072e-56c8-4f49-966c-18445c91eea2\" (UID: \"cc4d072e-56c8-4f49-966c-18445c91eea2\") " Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.262831 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-socket" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.263009 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-data" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.263315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.263196 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4d072e-56c8-4f49-966c-18445c91eea2-istiod-ca-cert" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:49.266218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.265368 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.266218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.265392 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc4d072e-56c8-4f49-966c-18445c91eea2-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.266218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.265406 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.266218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.265443 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.266218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.265459 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.327102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.327064 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-kube-api-access-8zx9q" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "kube-api-access-8zx9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.327302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.327207 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-envoy" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.327889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.327844 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-token" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.344466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.340605 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-podinfo" pod "cc4d072e-56c8-4f49-966c-18445c91eea2" (UID: "cc4d072e-56c8-4f49-966c-18445c91eea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:49.360962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.358869 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:25:49.364376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.364352 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:25:49.366991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.366966 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-socket\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.367115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367043 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istiod-ca-cert\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.367115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367099 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-data\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.367324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367267 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-podinfo\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.367383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367351 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-certs\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.367439 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367409 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-kube-api-access-fl7sg\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-kube-api-access-fl7sg\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.367500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367469 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-credential-socket\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.367500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367489 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istiod-ca-cert" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:49.367603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367555 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-token\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-token\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.367724 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-envoy\") pod \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\" (UID: \"97247502-e6a3-44b3-aef5-cd9a3c71dd64\") " Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368227 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-socket" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368560 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-certs" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368694 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zx9q\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-kube-api-access-8zx9q\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368714 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368729 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368742 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368756 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368768 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368781 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc4d072e-56c8-4f49-966c-18445c91eea2-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.368870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.368846 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-credential-socket" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.369753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.369719 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-data" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.371442 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.371413 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-envoy" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.372016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.371976 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-podinfo" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:49.372112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.372040 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-kube-api-access-fl7sg" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "kube-api-access-fl7sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.372565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.372531 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-token" pod "97247502-e6a3-44b3-aef5-cd9a3c71dd64" (UID: "97247502-e6a3-44b3-aef5-cd9a3c71dd64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.474178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.474138 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-envoy\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.474815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.474792 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-credential-socket\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.475384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.475366 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-data\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.475996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.475762 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-socket\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.475996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.475792 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-certs\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.476611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.475258 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-credential-socket" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.476740 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.475677 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-data" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.476857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.476393 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-socket" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.477002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.476582 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-certs" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.477275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.477255 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30a21199-1155-43cc-b36b-6215e6d3130b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a21199-1155-43cc-b36b-6215e6d3130b-istio-podinfo\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.477807 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.477789 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a21199-1155-43cc-b36b-6215e6d3130b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a21199-1155-43cc-b36b-6215e6d3130b-istiod-ca-cert\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.478411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.478393 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-kube-api-access-vtg5h\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-kube-api-access-vtg5h\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.478855 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.478830 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-istio-token\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-istio-token\") pod \"30a21199-1155-43cc-b36b-6215e6d3130b\" (UID: \"30a21199-1155-43cc-b36b-6215e6d3130b\") " Oct 30 18:25:49.479346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.478258 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a21199-1155-43cc-b36b-6215e6d3130b-istiod-ca-cert" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:49.480306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.480282 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7sg\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-kube-api-access-fl7sg\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.480439 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.480425 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.480534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.480521 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.480625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.480615 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30a21199-1155-43cc-b36b-6215e6d3130b-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.480710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.480700 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.480780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.480769 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.480856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.480845 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.481056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.480922 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.481174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.481160 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.481267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.481256 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.481353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.481343 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97247502-e6a3-44b3-aef5-cd9a3c71dd64-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.483223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.483202 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30a21199-1155-43cc-b36b-6215e6d3130b-istio-podinfo" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:49.508099 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.508045 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-envoy" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:49.511802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.511761 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-kube-api-access-vtg5h" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "kube-api-access-vtg5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.512919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.512849 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-istio-token" pod "30a21199-1155-43cc-b36b-6215e6d3130b" (UID: "30a21199-1155-43cc-b36b-6215e6d3130b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:49.616974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.583310 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30a21199-1155-43cc-b36b-6215e6d3130b-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.616974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.583341 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30a21199-1155-43cc-b36b-6215e6d3130b-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.616974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.583356 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtg5h\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-kube-api-access-vtg5h\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.616974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.583371 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30a21199-1155-43cc-b36b-6215e6d3130b-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:49.621176 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.620559 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:49.654890 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.654857 5624 generic.go:356] "Generic (PLEG): container finished" podID="30a21199-1155-43cc-b36b-6215e6d3130b" containerID="29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada" exitCode=0 Oct 30 18:25:49.655194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.655023 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" Oct 30 18:25:49.655586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.655046 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" event={"ID":"30a21199-1155-43cc-b36b-6215e6d3130b","Type":"ContainerDied","Data":"29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada"} Oct 30 18:25:49.655739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.655639 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d" event={"ID":"30a21199-1155-43cc-b36b-6215e6d3130b","Type":"ContainerDied","Data":"199af96dd577981d9a75b0e16be8620fdefb0230ef7dddc814074ee8ae15013b"} Oct 30 18:25:49.655739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.655680 5624 scope.go:122] "RemoveContainer" containerID="29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada" Oct 30 18:25:49.658166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.658140 5624 generic.go:356] "Generic (PLEG): container finished" podID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerID="255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c" exitCode=0 Oct 30 18:25:49.658266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.658239 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" event={"ID":"97247502-e6a3-44b3-aef5-cd9a3c71dd64","Type":"ContainerDied","Data":"255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c"} Oct 30 18:25:49.658339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.658282 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" event={"ID":"97247502-e6a3-44b3-aef5-cd9a3c71dd64","Type":"ContainerDied","Data":"b608d2f8ce0e304bf0d90764c8d2d924c55e7ef2f70598c004b6b26e99e41b67"} Oct 30 18:25:49.658398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.658360 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr" Oct 30 18:25:49.669391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.668282 5624 generic.go:356] "Generic (PLEG): container finished" podID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerID="897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d" exitCode=0 Oct 30 18:25:49.669391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.668406 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" event={"ID":"cc4d072e-56c8-4f49-966c-18445c91eea2","Type":"ContainerDied","Data":"897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d"} Oct 30 18:25:49.669391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.668450 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" event={"ID":"cc4d072e-56c8-4f49-966c-18445c91eea2","Type":"ContainerDied","Data":"60396a22644890f48f5370ac2efc8f8636847a61fda30ee234b879dba9c5a080"} Oct 30 18:25:49.669391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.668521 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx" Oct 30 18:25:49.675738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.675721 5624 scope.go:122] "RemoveContainer" containerID="29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada" Oct 30 18:25:49.679415 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:49.679390 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada\": container with ID starting with 29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada not found: ID does not exist" containerID="29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada" Oct 30 18:25:49.679570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.679543 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada"} err="failed to get container status \"29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada\": rpc error: code = NotFound desc = could not find container \"29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada\": container with ID starting with 29feb4029fdfa812041e793560fa9683681b813d9207156a4f7a1111a828cada not found: ID does not exist" Oct 30 18:25:49.680110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:49.680078 5624 scope.go:122] "RemoveContainer" containerID="255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c" Oct 30 18:25:50.072536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.058634 5624 scope.go:122] "RemoveContainer" containerID="255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c" Oct 30 18:25:50.072536 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:50.068346 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c\": container with ID starting with 255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c not found: ID does not exist" containerID="255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c" Oct 30 18:25:50.072536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.068385 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c"} err="failed to get container status \"255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c\": rpc error: code = NotFound desc = could not find container \"255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c\": container with ID starting with 255bd0aa2bee29eb611295a36a094501cb4b66917bb2652e3017861a08125d9c not found: ID does not exist" Oct 30 18:25:50.072536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.068414 5624 scope.go:122] "RemoveContainer" containerID="897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d" Oct 30 18:25:50.134629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.124613 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" podUID="13b40734-590f-4568-b812-c77303a52d56" containerName="payload-processing" containerID="cri-o://4551d941857a4fd7a01c32642a98b30d40486b3176292df7258bf8f6f2091978" gracePeriod=25 Oct 30 18:25:50.161135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.159011 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" podUID="0ffdb840-fd18-4ad2-8635-2234ab2069b0" containerName="payload-pre-processing" containerID="cri-o://783b130c6019ae2d1c7ce699417eaad2ac8529751b4a3304210ee5ec3330a9f7" gracePeriod=25 Oct 30 18:25:50.349804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.349775 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d"] Oct 30 18:25:50.476675 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:25:50.475301 5624 logging.go:55] [core] [Channel #2249 SubChannel #2251]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.124:50052", ServerName: "10.134.0.124:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.124:50052: connect: connection refused" Oct 30 18:25:50.491598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.482469 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-52v7d"] Oct 30 18:25:50.517953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.517905 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr"] Oct 30 18:25:50.568003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.557368 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-8rrfr"] Oct 30 18:25:50.568003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.559436 5624 scope.go:122] "RemoveContainer" containerID="82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae" Oct 30 18:25:50.568003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.565861 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx"] Oct 30 18:25:50.582107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.581957 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tkknx"] Oct 30 18:25:50.610231 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:50.610162 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:25:50.644527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.644497 5624 scope.go:122] "RemoveContainer" containerID="897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d" Oct 30 18:25:50.645109 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:50.645077 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d\": container with ID starting with 897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d not found: ID does not exist" containerID="897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d" Oct 30 18:25:50.645215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.645129 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d"} err="failed to get container status \"897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d\": rpc error: code = NotFound desc = could not find container \"897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d\": container with ID starting with 897d2fd6ad87042a77a3745eb052db61ee1ac57bc4d9775ec877f5b07ddcb31d not found: ID does not exist" Oct 30 18:25:50.645215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.645156 5624 scope.go:122] "RemoveContainer" containerID="82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae" Oct 30 18:25:50.645775 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:50.645746 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae\": container with ID starting with 82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae not found: ID does not exist" containerID="82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae" Oct 30 18:25:50.645871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.645799 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae"} err="failed to get container status \"82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae\": rpc error: code = NotFound desc = could not find container \"82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae\": container with ID starting with 82baf53f547489daab7d45cba9638223e70e764ff94e4ac3c1fefeeffaf839ae not found: ID does not exist" Oct 30 18:25:50.726422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.726390 5624 generic.go:356] "Generic (PLEG): container finished" podID="0ffdb840-fd18-4ad2-8635-2234ab2069b0" containerID="783b130c6019ae2d1c7ce699417eaad2ac8529751b4a3304210ee5ec3330a9f7" exitCode=0 Oct 30 18:25:50.741276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.741088 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" event={"ID":"0ffdb840-fd18-4ad2-8635-2234ab2069b0","Type":"ContainerDied","Data":"783b130c6019ae2d1c7ce699417eaad2ac8529751b4a3304210ee5ec3330a9f7"} Oct 30 18:25:50.836372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.782926 5624 generic.go:356] "Generic (PLEG): container finished" podID="13b40734-590f-4568-b812-c77303a52d56" containerID="4551d941857a4fd7a01c32642a98b30d40486b3176292df7258bf8f6f2091978" exitCode=0 Oct 30 18:25:50.836372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.783226 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" event={"ID":"13b40734-590f-4568-b812-c77303a52d56","Type":"ContainerDied","Data":"4551d941857a4fd7a01c32642a98b30d40486b3176292df7258bf8f6f2091978"} Oct 30 18:25:50.910154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.910122 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:25:50.941267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.941231 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b40734-590f-4568-b812-c77303a52d56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b40734-590f-4568-b812-c77303a52d56-plugins-config-volume\") pod \"13b40734-590f-4568-b812-c77303a52d56\" (UID: \"13b40734-590f-4568-b812-c77303a52d56\") " Oct 30 18:25:50.941468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.941450 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b40734-590f-4568-b812-c77303a52d56-kube-api-access-km2xb\" (UniqueName: \"kubernetes.io/projected/13b40734-590f-4568-b812-c77303a52d56-kube-api-access-km2xb\") pod \"13b40734-590f-4568-b812-c77303a52d56\" (UID: \"13b40734-590f-4568-b812-c77303a52d56\") " Oct 30 18:25:50.942382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.942348 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b40734-590f-4568-b812-c77303a52d56-plugins-config-volume" pod "13b40734-590f-4568-b812-c77303a52d56" (UID: "13b40734-590f-4568-b812-c77303a52d56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:50.943220 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.943196 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13b40734-590f-4568-b812-c77303a52d56-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:50.945036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:50.945009 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b40734-590f-4568-b812-c77303a52d56-kube-api-access-km2xb" pod "13b40734-590f-4568-b812-c77303a52d56" (UID: "13b40734-590f-4568-b812-c77303a52d56"). InnerVolumeSpecName "kube-api-access-km2xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:51.074843 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.068855 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:51.081325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.081049 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km2xb\" (UniqueName: \"kubernetes.io/projected/13b40734-590f-4568-b812-c77303a52d56-kube-api-access-km2xb\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:51.128057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.128032 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:25:51.222957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.220299 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" podUID="0ffdb840-fd18-4ad2-8635-2234ab2069b0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.124:50052\" within 1s: context deadline exceeded" Oct 30 18:25:51.283859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.283830 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ffdb840-fd18-4ad2-8635-2234ab2069b0-kube-api-access-j9m4b\" (UniqueName: \"kubernetes.io/projected/0ffdb840-fd18-4ad2-8635-2234ab2069b0-kube-api-access-j9m4b\") pod \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\" (UID: \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\") " Oct 30 18:25:51.284059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.283992 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ffdb840-fd18-4ad2-8635-2234ab2069b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffdb840-fd18-4ad2-8635-2234ab2069b0-plugins-config-volume\") pod \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\" (UID: \"0ffdb840-fd18-4ad2-8635-2234ab2069b0\") " Oct 30 18:25:51.284696 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.284669 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ffdb840-fd18-4ad2-8635-2234ab2069b0-plugins-config-volume" pod "0ffdb840-fd18-4ad2-8635-2234ab2069b0" (UID: "0ffdb840-fd18-4ad2-8635-2234ab2069b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:51.285057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.285035 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ffdb840-fd18-4ad2-8635-2234ab2069b0-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:51.287834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.287806 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ffdb840-fd18-4ad2-8635-2234ab2069b0-kube-api-access-j9m4b" pod "0ffdb840-fd18-4ad2-8635-2234ab2069b0" (UID: "0ffdb840-fd18-4ad2-8635-2234ab2069b0"). InnerVolumeSpecName "kube-api-access-j9m4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:51.330626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.330599 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:51.390005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.389552 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9m4b\" (UniqueName: \"kubernetes.io/projected/0ffdb840-fd18-4ad2-8635-2234ab2069b0-kube-api-access-j9m4b\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:51.806003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.803433 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:51.826336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.826310 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" Oct 30 18:25:51.827448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.827420 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" path="/var/lib/kubelet/pods/30a21199-1155-43cc-b36b-6215e6d3130b/volumes" Oct 30 18:25:51.828060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.828033 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" path="/var/lib/kubelet/pods/97247502-e6a3-44b3-aef5-cd9a3c71dd64/volumes" Oct 30 18:25:51.828169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.828160 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" Oct 30 18:25:51.829047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.829027 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" path="/var/lib/kubelet/pods/cc4d072e-56c8-4f49-966c-18445c91eea2/volumes" Oct 30 18:25:51.830015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.829994 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6" event={"ID":"13b40734-590f-4568-b812-c77303a52d56","Type":"ContainerDied","Data":"bcc4bf408b7c6b61809cc2751c0cd7eca5b54aab8e5c2d5e4fff1b10a541bc9d"} Oct 30 18:25:51.830123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.830034 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd" event={"ID":"0ffdb840-fd18-4ad2-8635-2234ab2069b0","Type":"ContainerDied","Data":"55f1067b89d51a8a529d8d3bba1e770f5ee62bea09ce6c06588fc1acf17e389d"} Oct 30 18:25:51.830123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.830058 5624 scope.go:122] "RemoveContainer" containerID="4551d941857a4fd7a01c32642a98b30d40486b3176292df7258bf8f6f2091978" Oct 30 18:25:51.848501 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.848483 5624 scope.go:122] "RemoveContainer" containerID="783b130c6019ae2d1c7ce699417eaad2ac8529751b4a3304210ee5ec3330a9f7" Oct 30 18:25:51.941041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.938107 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:51.947529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.947497 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd"] Oct 30 18:25:51.954187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.954154 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9af5623f-68cf6685c5-dg7sd"] Oct 30 18:25:51.982042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.982011 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6"] Oct 30 18:25:51.990397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:51.990361 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9af5623f-5854fb7bfb-4kwq6"] Oct 30 18:25:52.999796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:52.999693 5624 scope.go:122] "RemoveContainer" containerID="b4096edb54b80e0fbb40604e9788d453d7117a454bdddd89f6438c6449675f97" Oct 30 18:25:53.045437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:53.045396 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:53.796573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:53.796532 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ffdb840-fd18-4ad2-8635-2234ab2069b0" path="/var/lib/kubelet/pods/0ffdb840-fd18-4ad2-8635-2234ab2069b0/volumes" Oct 30 18:25:53.797155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:53.797133 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b40734-590f-4568-b812-c77303a52d56" path="/var/lib/kubelet/pods/13b40734-590f-4568-b812-c77303a52d56/volumes" Oct 30 18:25:55.373212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:55.373181 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:56.888449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:56.888414 5624 generic.go:356] "Generic (PLEG): container finished" podID="fea5d15c-4957-49c0-9739-f5383e23f424" containerID="1da3f66ff36bea22449d3ecd8175988bae0e3d9d8c90387d3ca0f55e744143fd" exitCode=0 Oct 30 18:25:56.888963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:56.888482 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" event={"ID":"fea5d15c-4957-49c0-9739-f5383e23f424","Type":"ContainerDied","Data":"1da3f66ff36bea22449d3ecd8175988bae0e3d9d8c90387d3ca0f55e744143fd"} Oct 30 18:25:56.888963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:56.888542 5624 scope.go:122] "RemoveContainer" containerID="cc434f884a702dd2d77b7578742773d3dd3d5fe5c964b602f959a2cff97c725a" Oct 30 18:25:56.889119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:56.889101 5624 scope.go:122] "RemoveContainer" containerID="1da3f66ff36bea22449d3ecd8175988bae0e3d9d8c90387d3ca0f55e744143fd" Oct 30 18:25:56.889549 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:25:56.889522 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr_openshift-ingress(fea5d15c-4957-49c0-9739-f5383e23f424)\"" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" Oct 30 18:25:57.881020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:57.880988 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2"] Oct 30 18:25:57.881222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:57.881196 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" containerID="cri-o://b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882" gracePeriod=30 Oct 30 18:25:58.150168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:58.150077 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.261033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:58.261000 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.521020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:58.514850 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:58.670625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:58.670581 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr"] Oct 30 18:25:58.787334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:58.787239 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7"] Oct 30 18:25:58.787508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:58.787465 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" containerID="cri-o://3d3a30f983dc3c6ce28482eea1d1290d6365c230dead46a3e9197a3f02d77f32" gracePeriod=30 Oct 30 18:25:59.213982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.213557 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:25:59.370730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.370660 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-istio-token\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-istio-token\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.370888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.370847 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fea5d15c-4957-49c0-9739-f5383e23f424-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea5d15c-4957-49c0-9739-f5383e23f424-istio-podinfo\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.371005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.370984 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-data\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.371076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371026 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-credential-socket\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.371076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371055 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-certs\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.371191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371145 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-envoy\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.371191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371165 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea5d15c-4957-49c0-9739-f5383e23f424-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea5d15c-4957-49c0-9739-f5383e23f424-istiod-ca-cert\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.371328 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371297 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-data" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:59.371404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371338 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-credential-socket" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:59.371491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371440 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-certs" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:59.371606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371584 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea5d15c-4957-49c0-9739-f5383e23f424-istiod-ca-cert" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:59.371659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371620 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-kube-api-access-gv8tg\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-kube-api-access-gv8tg\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.371712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371663 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-socket\") pod \"fea5d15c-4957-49c0-9739-f5383e23f424\" (UID: \"fea5d15c-4957-49c0-9739-f5383e23f424\") " Oct 30 18:25:59.371987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.371966 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-socket" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:59.372365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.372341 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.372365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.372365 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.372542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.372380 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.372542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.372392 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea5d15c-4957-49c0-9739-f5383e23f424-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.372542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.372399 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.373289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.373266 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fea5d15c-4957-49c0-9739-f5383e23f424-istio-podinfo" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:59.373493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.373471 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-istio-token" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:59.373567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.373474 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-envoy" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:59.373690 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.373674 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-kube-api-access-gv8tg" pod "fea5d15c-4957-49c0-9739-f5383e23f424" (UID: "fea5d15c-4957-49c0-9739-f5383e23f424"). InnerVolumeSpecName "kube-api-access-gv8tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:59.474035 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.474001 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea5d15c-4957-49c0-9739-f5383e23f424-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.474035 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.474026 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv8tg\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-kube-api-access-gv8tg\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.474035 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.474037 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea5d15c-4957-49c0-9739-f5383e23f424-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.474318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.474048 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea5d15c-4957-49c0-9739-f5383e23f424-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:25:59.475297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.475272 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:59.607975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.607925 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:59.718747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.718715 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:59.952307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.952274 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" Oct 30 18:25:59.952497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.952305 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr" event={"ID":"fea5d15c-4957-49c0-9739-f5383e23f424","Type":"ContainerDied","Data":"f05fb32340d97894e911163000e528fd9b22972e9008c0c58e6c2e2ec1f681df"} Oct 30 18:25:59.952497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:25:59.952379 5624 scope.go:122] "RemoveContainer" containerID="1da3f66ff36bea22449d3ecd8175988bae0e3d9d8c90387d3ca0f55e744143fd" Oct 30 18:26:00.045615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:00.045532 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr"] Oct 30 18:26:00.068271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:00.068239 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-ktlbr"] Oct 30 18:26:00.727897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:00.727864 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:00.852988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:00.852926 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:00.973008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:00.972982 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:01.795752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:01.795709 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" path="/var/lib/kubelet/pods/fea5d15c-4957-49c0-9739-f5383e23f424/volumes" Oct 30 18:26:02.080225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:02.080145 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:03.203029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.203002 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:26:03.281964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.281915 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-data\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.282173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.282081 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-podinfo\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.282173 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.282148 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-certs\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.282869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.282814 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-certs" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.283092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283071 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-socket\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.283195 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283165 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-credential-socket\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.283195 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283178 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-data" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.283320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283202 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-envoy\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.283320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283266 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-token\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-token\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.283428 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283398 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-kube-api-access-9jslw\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-kube-api-access-9jslw\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.283428 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283413 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-socket" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.283544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283512 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10cb470-5e0a-4def-806d-0e08450f10a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10cb470-5e0a-4def-806d-0e08450f10a2-istiod-ca-cert\") pod \"a10cb470-5e0a-4def-806d-0e08450f10a2\" (UID: \"a10cb470-5e0a-4def-806d-0e08450f10a2\") " Oct 30 18:26:03.283880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.283843 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-credential-socket" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.284334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.284298 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10cb470-5e0a-4def-806d-0e08450f10a2-istiod-ca-cert" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:03.285205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.285183 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10cb470-5e0a-4def-806d-0e08450f10a2-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.285422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.285403 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.285547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.285425 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.285547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.285440 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.285547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.285452 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.285547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.285364 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-podinfo" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:03.288062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.288037 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-envoy" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:03.288226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.288166 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-kube-api-access-9jslw" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "kube-api-access-9jslw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:03.288714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.288691 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-token" pod "a10cb470-5e0a-4def-806d-0e08450f10a2" (UID: "a10cb470-5e0a-4def-806d-0e08450f10a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:03.387197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.387164 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.387197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.387194 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.387197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.387203 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:03.387435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:03.387212 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jslw\" (UniqueName: \"kubernetes.io/projected/a10cb470-5e0a-4def-806d-0e08450f10a2-kube-api-access-9jslw\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.007455 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.007117 5624 generic.go:356] "Generic (PLEG): container finished" podID="1470d092-0818-4480-a80a-00b91adbeb4f" containerID="3d3a30f983dc3c6ce28482eea1d1290d6365c230dead46a3e9197a3f02d77f32" exitCode=0 Oct 30 18:26:04.007455 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.007211 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" event={"ID":"1470d092-0818-4480-a80a-00b91adbeb4f","Type":"ContainerDied","Data":"3d3a30f983dc3c6ce28482eea1d1290d6365c230dead46a3e9197a3f02d77f32"} Oct 30 18:26:04.012007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.009534 5624 generic.go:356] "Generic (PLEG): container finished" podID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerID="b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882" exitCode=0 Oct 30 18:26:04.012007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.009723 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" event={"ID":"a10cb470-5e0a-4def-806d-0e08450f10a2","Type":"ContainerDied","Data":"b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882"} Oct 30 18:26:04.012007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.009758 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" event={"ID":"a10cb470-5e0a-4def-806d-0e08450f10a2","Type":"ContainerDied","Data":"88e5df1dd892012347f91c97752ede7a417348ca9857685695970340a20b745a"} Oct 30 18:26:04.012007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.009785 5624 scope.go:122] "RemoveContainer" containerID="b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882" Oct 30 18:26:04.012007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.010027 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2" Oct 30 18:26:04.027734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.027711 5624 scope.go:122] "RemoveContainer" containerID="b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882" Oct 30 18:26:04.028076 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:26:04.028058 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882\": container with ID starting with b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882 not found: ID does not exist" containerID="b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882" Oct 30 18:26:04.028161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.028083 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882"} err="failed to get container status \"b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882\": rpc error: code = NotFound desc = could not find container \"b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882\": container with ID starting with b7f159a7d99812f928119edcee26604c1ac310b852b59c551afc7ca8e1aa3882 not found: ID does not exist" Oct 30 18:26:04.033895 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.032529 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:04.108760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.108641 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2"] Oct 30 18:26:04.136861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.136792 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-lzmg2"] Oct 30 18:26:04.181537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.181510 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:26:04.307376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.307344 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-envoy\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.307897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.307393 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1470d092-0818-4480-a80a-00b91adbeb4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1470d092-0818-4480-a80a-00b91adbeb4f-istiod-ca-cert\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.307897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.307505 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1470d092-0818-4480-a80a-00b91adbeb4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1470d092-0818-4480-a80a-00b91adbeb4f-istio-podinfo\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.307897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.307600 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-data\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.308103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.307862 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-data" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.308103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.307872 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1470d092-0818-4480-a80a-00b91adbeb4f-istiod-ca-cert" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:04.308103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.308005 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-istio-token\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-istio-token\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.308103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.308084 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-certs\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.308318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.308208 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-kube-api-access-2jqhn\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-kube-api-access-2jqhn\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.308318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.308235 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-socket\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.308318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.308288 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-credential-socket\") pod \"1470d092-0818-4480-a80a-00b91adbeb4f\" (UID: \"1470d092-0818-4480-a80a-00b91adbeb4f\") " Oct 30 18:26:04.308493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.308406 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-certs" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.308896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.308870 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-socket" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.309158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.309136 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-credential-socket" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.309416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.309396 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1470d092-0818-4480-a80a-00b91adbeb4f-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.309488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.309422 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.309488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.309437 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.309488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.309451 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.309488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.309465 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.309954 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.309919 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-envoy" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:04.310157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.310136 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1470d092-0818-4480-a80a-00b91adbeb4f-istio-podinfo" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:04.310495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.310467 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-istio-token" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:04.311127 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.311104 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-kube-api-access-2jqhn" pod "1470d092-0818-4480-a80a-00b91adbeb4f" (UID: "1470d092-0818-4480-a80a-00b91adbeb4f"). InnerVolumeSpecName "kube-api-access-2jqhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:04.411606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.411575 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.411606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.411609 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jqhn\" (UniqueName: \"kubernetes.io/projected/1470d092-0818-4480-a80a-00b91adbeb4f-kube-api-access-2jqhn\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.411808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.411624 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1470d092-0818-4480-a80a-00b91adbeb4f-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:04.411808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:04.411638 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1470d092-0818-4480-a80a-00b91adbeb4f-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:05.021530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.021502 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" Oct 30 18:26:05.021530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.021521 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7" event={"ID":"1470d092-0818-4480-a80a-00b91adbeb4f","Type":"ContainerDied","Data":"6ee95902cb4761d34ba6b0dff76a995fe0a7b2d1f8cccb3753de6e51d22eb142"} Oct 30 18:26:05.021789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.021588 5624 scope.go:122] "RemoveContainer" containerID="3d3a30f983dc3c6ce28482eea1d1290d6365c230dead46a3e9197a3f02d77f32" Oct 30 18:26:05.096793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.096763 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7"] Oct 30 18:26:05.109290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.109263 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-qmfr7"] Oct 30 18:26:05.672363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.672325 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5"] Oct 30 18:26:05.782684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.782650 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:05.801554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.801525 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" path="/var/lib/kubelet/pods/1470d092-0818-4480-a80a-00b91adbeb4f/volumes" Oct 30 18:26:05.802436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.802409 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" path="/var/lib/kubelet/pods/a10cb470-5e0a-4def-806d-0e08450f10a2/volumes" Oct 30 18:26:05.882676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.882642 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:05.982302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:05.982225 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:07.602586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:07.602551 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:09.241547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:09.241514 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:10.673793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:10.673756 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" podUID="f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" containerName="payload-pre-processing" containerID="cri-o://e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23" gracePeriod=25 Oct 30 18:26:10.864469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:10.864432 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:10.963497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:10.963417 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:11.018803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.018776 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:26:11.115964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.115581 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:11.120883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.120144 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-kube-api-access-wpmkc\" (UniqueName: \"kubernetes.io/projected/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-kube-api-access-wpmkc\") pod \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\" (UID: \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\") " Oct 30 18:26:11.120883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.120283 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-plugins-config-volume\") pod \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\" (UID: \"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2\") " Oct 30 18:26:11.127238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.121505 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-plugins-config-volume" pod "f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" (UID: "f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:11.145332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.145285 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-kube-api-access-wpmkc" pod "f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" (UID: "f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2"). InnerVolumeSpecName "kube-api-access-wpmkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.151694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.149331 5624 generic.go:356] "Generic (PLEG): container finished" podID="f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" containerID="e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23" exitCode=0 Oct 30 18:26:11.151694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.149410 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" event={"ID":"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2","Type":"ContainerDied","Data":"e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23"} Oct 30 18:26:11.151694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.149620 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" event={"ID":"f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2","Type":"ContainerDied","Data":"d12e82f96cf15235e2119ac7f2b857b4cbc858a61e3cd8dd4abd6f9574dfedda"} Oct 30 18:26:11.151694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.149670 5624 scope.go:122] "RemoveContainer" containerID="e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23" Oct 30 18:26:11.151694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.149874 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5" Oct 30 18:26:11.183294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.183165 5624 scope.go:122] "RemoveContainer" containerID="e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23" Oct 30 18:26:11.183608 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:26:11.183582 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23\": container with ID starting with e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23 not found: ID does not exist" containerID="e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23" Oct 30 18:26:11.183709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.183626 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23"} err="failed to get container status \"e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23\": rpc error: code = NotFound desc = could not find container \"e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23\": container with ID starting with e5d8361646899a7b7a230095c6a69ce142ca8ff14a3d28449c85bc450fd09d23 not found: ID does not exist" Oct 30 18:26:11.216432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.216339 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5"] Oct 30 18:26:11.222857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.222836 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpmkc\" (UniqueName: \"kubernetes.io/projected/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-kube-api-access-wpmkc\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.222857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.222857 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.224893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.224871 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-295497-84b9b64fff-vn8z5"] Oct 30 18:26:11.798755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:11.798723 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" path="/var/lib/kubelet/pods/f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2/volumes" Oct 30 18:26:12.917157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:12.917118 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:13.019653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:13.019616 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:14.744187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:14.744155 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:16.474858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:16.474819 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:17.949132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:17.949094 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df"] Oct 30 18:26:17.949642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:17.949391 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" podUID="4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" containerName="maas-api" containerID="cri-o://6c02206141a75b56837587431fcd77b4d570aee3741118ee7921da10c1748c95" gracePeriod=30 Oct 30 18:26:17.999008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:17.998976 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:18.508109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.507706 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:26:18.555917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.530401 5624 generic.go:356] "Generic (PLEG): container finished" podID="4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" containerID="6c02206141a75b56837587431fcd77b4d570aee3741118ee7921da10c1748c95" exitCode=0 Oct 30 18:26:18.555917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.530515 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" event={"ID":"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d","Type":"ContainerDied","Data":"6c02206141a75b56837587431fcd77b4d570aee3741118ee7921da10c1748c95"} Oct 30 18:26:18.555917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.530554 5624 scope.go:122] "RemoveContainer" containerID="6c02206141a75b56837587431fcd77b4d570aee3741118ee7921da10c1748c95" Oct 30 18:26:18.555917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.530681 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df" Oct 30 18:26:18.637286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.588888 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-maas-api-tls\") pod \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " Oct 30 18:26:18.637286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.589013 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-openshift-service-ca\") pod \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " Oct 30 18:26:18.637286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.589159 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-metrics-certs\") pod \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " Oct 30 18:26:18.637286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.589218 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-kube-api-access-2ggjr\" (UniqueName: \"kubernetes.io/projected/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-kube-api-access-2ggjr\") pod \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\" (UID: \"4f7a7f1d-20bd-41f1-b298-14543a6d2d1d\") " Oct 30 18:26:18.637286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.610332 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-openshift-service-ca" pod "4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" (UID: "4f7a7f1d-20bd-41f1-b298-14543a6d2d1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:18.661468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.647725 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-kube-api-access-2ggjr" pod "4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" (UID: "4f7a7f1d-20bd-41f1-b298-14543a6d2d1d"). InnerVolumeSpecName "kube-api-access-2ggjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:18.674017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.662963 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-metrics-certs" pod "4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" (UID: "4f7a7f1d-20bd-41f1-b298-14543a6d2d1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:18.714027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.713614 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:18.714027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.713646 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:18.714027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.713662 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ggjr\" (UniqueName: \"kubernetes.io/projected/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-kube-api-access-2ggjr\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:18.737012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.725003 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-maas-api-tls" pod "4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" (UID: "4f7a7f1d-20bd-41f1-b298-14543a6d2d1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:18.815013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.814923 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:18.868420 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.868383 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:18.900016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.893636 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df"] Oct 30 18:26:18.900016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.899687 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f8709f-698b768567-kj8df"] Oct 30 18:26:18.963883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:18.963849 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:19.063456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:19.063419 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:19.496008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:19.492498 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:19.590037 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:19.590001 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:19.796053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:19.795972 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" path="/var/lib/kubelet/pods/4f7a7f1d-20bd-41f1-b298-14543a6d2d1d/volumes" Oct 30 18:26:19.916384 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:26:19.916347 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:26:20.841245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:20.841210 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:21.854563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:21.854519 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm"] Oct 30 18:26:21.940956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:21.940113 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:26:22.035988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:22.019658 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp"] Oct 30 18:26:22.177023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:22.175764 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:22.274096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:22.274062 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:23.212029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:23.211914 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:25.452892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:25.452856 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:26.158946 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:26.147711 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w"] Oct 30 18:26:26.160154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:26.160125 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" containerID="cri-o://ad8e94c5a5bf4cd3d35a8732a7cf746502d07a2f00af6478e9d64fb34ef121ca" gracePeriod=30 Oct 30 18:26:26.659617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:26.659473 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:26.797306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:26.797237 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:26.834638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:26.834595 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:26.915019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:26.835332 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:26:26.927068 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:26.925864 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:26.927068 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:26.835384 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:26:27.011109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:27.011066 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:27.053976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:27.052055 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" podUID="6899f054-fc85-459e-a1a7-9dee926b2c8c" containerName="payload-pre-processing" containerID="cri-o://f58950d044941ac00904e7ab1452f614f36b98bfd51ff4d68f4ca1cb3bc69180" gracePeriod=25 Oct 30 18:26:27.180018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:27.169665 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:27.220958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:27.199093 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:27.374056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:27.338101 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Oct 30 18:26:27.424953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:27.424050 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:27.612054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:27.612007 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" podUID="67687132-9d0b-4ebe-b344-3881f9369509" containerName="payload-processing" containerID="cri-o://87a04391c907d3c7f82437215de10c9acd1c364244db569696ac56c1f753deb8" gracePeriod=25 Oct 30 18:26:27.810393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:27.809964 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:26:28.066699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.063391 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:26:28.066699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.063669 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:28.066699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.063836 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 30 18:26:28.117998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.117957 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:26:28.135291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.133058 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:28.167580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.167382 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.294762 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.294826 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.294862 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.294895 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.432026 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.433297 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.438953 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.439141 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:28.474812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.439437 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:26:28.573995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.573954 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:26:28.841015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.837049 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:28.900019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.896506 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:26:28.994311 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:28.942308 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:26:29.066380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.066344 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:29.067159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.067129 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:26:29.067598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.067575 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:26:29.132888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.128031 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:26:29.181064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.177563 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" event={"ID":"67687132-9d0b-4ebe-b344-3881f9369509","Type":"ContainerDied","Data":"87a04391c907d3c7f82437215de10c9acd1c364244db569696ac56c1f753deb8"} Oct 30 18:26:29.181064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.177680 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:26:29.182383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.182355 5624 generic.go:356] "Generic (PLEG): container finished" podID="67687132-9d0b-4ebe-b344-3881f9369509" containerID="87a04391c907d3c7f82437215de10c9acd1c364244db569696ac56c1f753deb8" exitCode=0 Oct 30 18:26:29.273474 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.271428 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:26:29.280231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.280202 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:26:29.283175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.283144 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:26:29.283451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.283421 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:26:29.351683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.343884 5624 generic.go:356] "Generic (PLEG): container finished" podID="6899f054-fc85-459e-a1a7-9dee926b2c8c" containerID="f58950d044941ac00904e7ab1452f614f36b98bfd51ff4d68f4ca1cb3bc69180" exitCode=0 Oct 30 18:26:29.351683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.344347 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" event={"ID":"6899f054-fc85-459e-a1a7-9dee926b2c8c","Type":"ContainerDied","Data":"f58950d044941ac00904e7ab1452f614f36b98bfd51ff4d68f4ca1cb3bc69180"} Oct 30 18:26:29.351683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.344511 5624 scope.go:122] "RemoveContainer" containerID="f58950d044941ac00904e7ab1452f614f36b98bfd51ff4d68f4ca1cb3bc69180" Oct 30 18:26:29.432771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.432739 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:26:29.529976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.527463 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6899f054-fc85-459e-a1a7-9dee926b2c8c-kube-api-access-6h2dg\" (UniqueName: \"kubernetes.io/projected/6899f054-fc85-459e-a1a7-9dee926b2c8c-kube-api-access-6h2dg\") pod \"6899f054-fc85-459e-a1a7-9dee926b2c8c\" (UID: \"6899f054-fc85-459e-a1a7-9dee926b2c8c\") " Oct 30 18:26:29.549017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.532748 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6899f054-fc85-459e-a1a7-9dee926b2c8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6899f054-fc85-459e-a1a7-9dee926b2c8c-plugins-config-volume\") pod \"6899f054-fc85-459e-a1a7-9dee926b2c8c\" (UID: \"6899f054-fc85-459e-a1a7-9dee926b2c8c\") " Oct 30 18:26:29.565721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.565693 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:26:29.566503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.566466 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6899f054-fc85-459e-a1a7-9dee926b2c8c-plugins-config-volume" pod "6899f054-fc85-459e-a1a7-9dee926b2c8c" (UID: "6899f054-fc85-459e-a1a7-9dee926b2c8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:29.574865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.574838 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6899f054-fc85-459e-a1a7-9dee926b2c8c-kube-api-access-6h2dg" pod "6899f054-fc85-459e-a1a7-9dee926b2c8c" (UID: "6899f054-fc85-459e-a1a7-9dee926b2c8c"). InnerVolumeSpecName "kube-api-access-6h2dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:29.634000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.633427 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:29.653051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.653020 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67687132-9d0b-4ebe-b344-3881f9369509-kube-api-access-kczfp\" (UniqueName: \"kubernetes.io/projected/67687132-9d0b-4ebe-b344-3881f9369509-kube-api-access-kczfp\") pod \"67687132-9d0b-4ebe-b344-3881f9369509\" (UID: \"67687132-9d0b-4ebe-b344-3881f9369509\") " Oct 30 18:26:29.653692 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.653670 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67687132-9d0b-4ebe-b344-3881f9369509-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67687132-9d0b-4ebe-b344-3881f9369509-plugins-config-volume\") pod \"67687132-9d0b-4ebe-b344-3881f9369509\" (UID: \"67687132-9d0b-4ebe-b344-3881f9369509\") " Oct 30 18:26:29.655315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.655232 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2dg\" (UniqueName: \"kubernetes.io/projected/6899f054-fc85-459e-a1a7-9dee926b2c8c-kube-api-access-6h2dg\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.655504 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.655464 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6899f054-fc85-459e-a1a7-9dee926b2c8c-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.655661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.654229 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67687132-9d0b-4ebe-b344-3881f9369509-plugins-config-volume" pod "67687132-9d0b-4ebe-b344-3881f9369509" (UID: "67687132-9d0b-4ebe-b344-3881f9369509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:29.664735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.664699 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67687132-9d0b-4ebe-b344-3881f9369509-kube-api-access-kczfp" pod "67687132-9d0b-4ebe-b344-3881f9369509" (UID: "67687132-9d0b-4ebe-b344-3881f9369509"). InnerVolumeSpecName "kube-api-access-kczfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:29.777091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.774306 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kczfp\" (UniqueName: \"kubernetes.io/projected/67687132-9d0b-4ebe-b344-3881f9369509-kube-api-access-kczfp\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:29.791048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:29.790998 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67687132-9d0b-4ebe-b344-3881f9369509-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:30.080452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.080418 5624 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:26:30.080908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.080487 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:26:30.357169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.357052 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" Oct 30 18:26:30.357416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.357052 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm" event={"ID":"67687132-9d0b-4ebe-b344-3881f9369509","Type":"ContainerDied","Data":"29db87bb6a7ecbb07fdbe296a7a464ba54080446a50674691f5574c6e2eafe30"} Oct 30 18:26:30.357522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.357464 5624 scope.go:122] "RemoveContainer" containerID="87a04391c907d3c7f82437215de10c9acd1c364244db569696ac56c1f753deb8" Oct 30 18:26:30.358629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.358600 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" Oct 30 18:26:30.358864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.358837 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp" event={"ID":"6899f054-fc85-459e-a1a7-9dee926b2c8c","Type":"ContainerDied","Data":"cf3cc26ef291db7fae083ad7cf26b5397b6a2b126b50778471ad2aa00a438f54"} Oct 30 18:26:30.421754 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.421720 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp"] Oct 30 18:26:30.434989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.434958 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fb0abbf7-7f95bbc587-qt8wp"] Oct 30 18:26:30.478520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.478471 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm"] Oct 30 18:26:30.479825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:30.479797 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fb0abbf7-65697d6b56-8l5hm"] Oct 30 18:26:31.379434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.379390 5624 generic.go:356] "Generic (PLEG): container finished" podID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerID="ad8e94c5a5bf4cd3d35a8732a7cf746502d07a2f00af6478e9d64fb34ef121ca" exitCode=0 Oct 30 18:26:31.379841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.379595 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" event={"ID":"ac309921-c2c5-4c6a-8635-79dea80cfadf","Type":"ContainerDied","Data":"ad8e94c5a5bf4cd3d35a8732a7cf746502d07a2f00af6478e9d64fb34ef121ca"} Oct 30 18:26:31.379841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.379658 5624 scope.go:122] "RemoveContainer" containerID="e6c0619e00ae7c0869b255b050920b47562d7bf2e5f0ca002d42bdebb0acd96e" Oct 30 18:26:31.394540 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.394506 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:31.504379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.503887 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:26:31.670670 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.670584 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-podinfo\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.670670 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.670646 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-kube-api-access-68n5b\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-kube-api-access-68n5b\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.670996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.670693 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-token\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-token\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.670996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.670840 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-credential-socket\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.670996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.670947 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac309921-c2c5-4c6a-8635-79dea80cfadf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac309921-c2c5-4c6a-8635-79dea80cfadf-istiod-ca-cert\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.671371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.671010 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-envoy\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.671371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.671047 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-data\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.671371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.671078 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-socket\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.671371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.671197 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-certs\") pod \"ac309921-c2c5-4c6a-8635-79dea80cfadf\" (UID: \"ac309921-c2c5-4c6a-8635-79dea80cfadf\") " Oct 30 18:26:31.672319 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.672285 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-credential-socket" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:31.674135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.673556 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac309921-c2c5-4c6a-8635-79dea80cfadf-istiod-ca-cert" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:31.677815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.677786 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-data" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:31.678313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.678289 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-certs" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:31.678682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.678650 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-socket" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:31.679221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.679198 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-podinfo" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:31.684898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.684858 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.687725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.686114 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-token" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:31.687725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.686431 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-kube-api-access-68n5b" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "kube-api-access-68n5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:31.687725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.686551 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-envoy" pod "ac309921-c2c5-4c6a-8635-79dea80cfadf" (UID: "ac309921-c2c5-4c6a-8635-79dea80cfadf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:31.786592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.786489 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.786592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.786551 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68n5b\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-kube-api-access-68n5b\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.786592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.786564 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.786592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.786578 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac309921-c2c5-4c6a-8635-79dea80cfadf-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.786592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.786590 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.786875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.786602 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.786875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.786614 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.786875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.786625 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac309921-c2c5-4c6a-8635-79dea80cfadf-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.795735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.795689 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67687132-9d0b-4ebe-b344-3881f9369509" path="/var/lib/kubelet/pods/67687132-9d0b-4ebe-b344-3881f9369509/volumes" Oct 30 18:26:31.796372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:31.796349 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6899f054-fc85-459e-a1a7-9dee926b2c8c" path="/var/lib/kubelet/pods/6899f054-fc85-459e-a1a7-9dee926b2c8c/volumes" Oct 30 18:26:32.433485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:32.432296 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" event={"ID":"ac309921-c2c5-4c6a-8635-79dea80cfadf","Type":"ContainerDied","Data":"e05ddd6d8017905a68859a2b61fa1f381e39ac80492b109fe4a31b538c98af37"} Oct 30 18:26:32.433485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:32.432362 5624 scope.go:122] "RemoveContainer" containerID="ad8e94c5a5bf4cd3d35a8732a7cf746502d07a2f00af6478e9d64fb34ef121ca" Oct 30 18:26:32.433485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:32.432492 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w" Oct 30 18:26:32.564970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:32.563028 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w"] Oct 30 18:26:32.571532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:32.571497 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-lsl2w"] Oct 30 18:26:33.043152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.042947 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:33.190748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.190718 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d"] Oct 30 18:26:33.191202 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.191168 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" podUID="ffa8cc09-b308-4348-af04-5773d5134665" containerName="maas-api" containerID="cri-o://97dc996afcb9b67f5197e84052ed231f0d35daea3941f3f575bab27bc5d36b4f" gracePeriod=30 Oct 30 18:26:33.499635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.499414 5624 generic.go:356] "Generic (PLEG): container finished" podID="ffa8cc09-b308-4348-af04-5773d5134665" containerID="97dc996afcb9b67f5197e84052ed231f0d35daea3941f3f575bab27bc5d36b4f" exitCode=0 Oct 30 18:26:33.499635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.499570 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" event={"ID":"ffa8cc09-b308-4348-af04-5773d5134665","Type":"ContainerDied","Data":"97dc996afcb9b67f5197e84052ed231f0d35daea3941f3f575bab27bc5d36b4f"} Oct 30 18:26:33.731425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.728541 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:33.831060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.823833 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" path="/var/lib/kubelet/pods/ac309921-c2c5-4c6a-8635-79dea80cfadf/volumes" Oct 30 18:26:33.850711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.850476 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:33.882908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.882872 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:26:33.958838 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:33.953485 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:34.028201 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.028168 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa8cc09-b308-4348-af04-5773d5134665-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8cc09-b308-4348-af04-5773d5134665-openshift-service-ca\") pod \"ffa8cc09-b308-4348-af04-5773d5134665\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " Oct 30 18:26:34.028397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.028371 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls\") pod \"ffa8cc09-b308-4348-af04-5773d5134665\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " Oct 30 18:26:34.028478 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.028409 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-metrics-certs\") pod \"ffa8cc09-b308-4348-af04-5773d5134665\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " Oct 30 18:26:34.028478 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.028457 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa8cc09-b308-4348-af04-5773d5134665-kube-api-access-5txvn\" (UniqueName: \"kubernetes.io/projected/ffa8cc09-b308-4348-af04-5773d5134665-kube-api-access-5txvn\") pod \"ffa8cc09-b308-4348-af04-5773d5134665\" (UID: \"ffa8cc09-b308-4348-af04-5773d5134665\") " Oct 30 18:26:34.028592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.028557 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa8cc09-b308-4348-af04-5773d5134665-openshift-service-ca" pod "ffa8cc09-b308-4348-af04-5773d5134665" (UID: "ffa8cc09-b308-4348-af04-5773d5134665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:34.029230 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.029205 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa8cc09-b308-4348-af04-5773d5134665-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:34.031327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.031227 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-metrics-certs" pod "ffa8cc09-b308-4348-af04-5773d5134665" (UID: "ffa8cc09-b308-4348-af04-5773d5134665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:34.031327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.031270 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls" pod "ffa8cc09-b308-4348-af04-5773d5134665" (UID: "ffa8cc09-b308-4348-af04-5773d5134665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:34.032797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.032735 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa8cc09-b308-4348-af04-5773d5134665-kube-api-access-5txvn" pod "ffa8cc09-b308-4348-af04-5773d5134665" (UID: "ffa8cc09-b308-4348-af04-5773d5134665"). InnerVolumeSpecName "kube-api-access-5txvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:34.134955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.130820 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txvn\" (UniqueName: \"kubernetes.io/projected/ffa8cc09-b308-4348-af04-5773d5134665-kube-api-access-5txvn\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:34.134955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.130855 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:34.134955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.130870 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffa8cc09-b308-4348-af04-5773d5134665-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:34.353429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.353340 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:34.522968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.522018 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" Oct 30 18:26:34.524624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.524098 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d" event={"ID":"ffa8cc09-b308-4348-af04-5773d5134665","Type":"ContainerDied","Data":"a56e5eb741bf733477201e9dec4f3f408ce1c8af533f2705c338da4ee89eb716"} Oct 30 18:26:34.524624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.524168 5624 scope.go:122] "RemoveContainer" containerID="97dc996afcb9b67f5197e84052ed231f0d35daea3941f3f575bab27bc5d36b4f" Oct 30 18:26:34.594711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.594675 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d"] Oct 30 18:26:34.602909 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:34.602882 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fb0abbf7-8dbf49df8-2xl2d"] Oct 30 18:26:35.660712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:35.660673 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:35.952539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:35.802220 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa8cc09-b308-4348-af04-5773d5134665" path="/var/lib/kubelet/pods/ffa8cc09-b308-4348-af04-5773d5134665/volumes" Oct 30 18:26:36.545569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:36.545540 5624 generic.go:356] "Generic (PLEG): container finished" podID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerID="0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f" exitCode=0 Oct 30 18:26:36.545756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:36.545615 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" event={"ID":"9e3d9000-f1f2-450b-9da7-c1de20b34b77","Type":"ContainerDied","Data":"0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f"} Oct 30 18:26:36.546080 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:36.546065 5624 scope.go:122] "RemoveContainer" containerID="0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f" Oct 30 18:26:36.547910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:36.547883 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:26:36.548005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:36.547970 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:26:36.548045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:36.548007 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:26:37.246508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:37.246479 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:37.555854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:37.555774 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" event={"ID":"9e3d9000-f1f2-450b-9da7-c1de20b34b77","Type":"ContainerStarted","Data":"4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711"} Oct 30 18:26:38.122676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:38.122639 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:26:38.123794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:38.123763 5624 patch_prober.go:29] interesting pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:26:38.123977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:38.123824 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:26:38.324793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:38.316910 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:38.443009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:38.442755 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:38.899964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:38.899231 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:39.124020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:39.123984 5624 patch_prober.go:29] interesting pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:26:39.135982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:39.134307 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:26:39.407438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:39.407401 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:39.815164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:39.815084 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:39.988006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:39.976689 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:40.122784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:40.122749 5624 patch_prober.go:29] interesting pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:26:40.123039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:40.122807 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:26:40.816170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:40.815904 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:41.123419 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:41.123390 5624 patch_prober.go:29] interesting pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:26:41.123603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:41.123445 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:26:42.123100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:42.123063 5624 patch_prober.go:29] interesting pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:26:42.123516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:42.123158 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:26:42.125684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:42.125633 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:42.127921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:42.127894 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9"] Oct 30 18:26:42.128194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:42.128149 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" containerID="cri-o://263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b" gracePeriod=30 Oct 30 18:26:42.463999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:42.463900 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:43.123253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:43.123200 5624 patch_prober.go:29] interesting pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 18:26:43.123680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:43.123267 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 18:26:43.436621 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:43.436535 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:26:44.128568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:44.128537 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:26:44.130165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:44.130139 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:26:44.245194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:44.245142 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:44.386338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:44.386213 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:46.032778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:46.032742 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:47.454534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.454510 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:26:47.553767 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.553730 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-envoy\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.553974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.553791 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-certs\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.553974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.553865 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-credential-socket\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.553974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.553947 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-data\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.554136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.553992 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aadedad1-fea9-471c-bd58-2372d6c91bb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadedad1-fea9-471c-bd58-2372d6c91bb8-istiod-ca-cert\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.554136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.554021 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-kube-api-access-6xm7p\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-kube-api-access-6xm7p\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.554265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.554239 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-podinfo\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.554318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.554296 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-token\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-token\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.554468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.554451 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-socket\") pod \"aadedad1-fea9-471c-bd58-2372d6c91bb8\" (UID: \"aadedad1-fea9-471c-bd58-2372d6c91bb8\") " Oct 30 18:26:47.555358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.554448 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadedad1-fea9-471c-bd58-2372d6c91bb8-istiod-ca-cert" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:47.555358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.554589 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-data" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:47.555358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.554830 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-socket" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:47.555358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.555269 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.555358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.555291 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.555358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.555306 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aadedad1-fea9-471c-bd58-2372d6c91bb8-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.556023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.555985 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-certs" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:47.556522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.556430 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-envoy" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:47.556738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.556710 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-credential-socket" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:47.556913 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.556887 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-podinfo" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:47.557388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.557362 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-token" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:47.558315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.558286 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-kube-api-access-6xm7p" pod "aadedad1-fea9-471c-bd58-2372d6c91bb8" (UID: "aadedad1-fea9-471c-bd58-2372d6c91bb8"). InnerVolumeSpecName "kube-api-access-6xm7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:47.656709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.656675 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.656709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.656704 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.656709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.656714 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.656988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.656723 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.656988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.656733 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aadedad1-fea9-471c-bd58-2372d6c91bb8-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.656988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.656743 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xm7p\" (UniqueName: \"kubernetes.io/projected/aadedad1-fea9-471c-bd58-2372d6c91bb8-kube-api-access-6xm7p\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:26:47.742959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.742854 5624 generic.go:356] "Generic (PLEG): container finished" podID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerID="263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b" exitCode=0 Oct 30 18:26:47.742959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.742896 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" Oct 30 18:26:47.742959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.742919 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" event={"ID":"aadedad1-fea9-471c-bd58-2372d6c91bb8","Type":"ContainerDied","Data":"263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b"} Oct 30 18:26:47.743224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.743014 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9" event={"ID":"aadedad1-fea9-471c-bd58-2372d6c91bb8","Type":"ContainerDied","Data":"819b1c9e1f2935b1a2dfa6ddea97b7ff361adbe0e5bd0564381732850f8e8926"} Oct 30 18:26:47.743224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.743044 5624 scope.go:122] "RemoveContainer" containerID="263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b" Oct 30 18:26:47.744905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.744871 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:47.755510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.755488 5624 scope.go:122] "RemoveContainer" containerID="263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b" Oct 30 18:26:47.755777 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:26:47.755754 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b\": container with ID starting with 263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b not found: ID does not exist" containerID="263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b" Oct 30 18:26:47.755826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.755782 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b"} err="failed to get container status \"263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b\": rpc error: code = NotFound desc = could not find container \"263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b\": container with ID starting with 263cbd257019e867f7d31f48b735d3ccb0a44c2dc6f8631410a704bb1a7cea1b not found: ID does not exist" Oct 30 18:26:47.832959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.822878 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9"] Oct 30 18:26:47.841404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:47.833352 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-rf9l9"] Oct 30 18:26:48.872363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:48.871658 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:49.432416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:49.432386 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:49.793539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:49.793456 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" path="/var/lib/kubelet/pods/aadedad1-fea9-471c-bd58-2372d6c91bb8/volumes" Oct 30 18:26:50.058647 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:26:50.058547 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:26:50.156011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:50.155879 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:50.697968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:50.687328 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:51.115880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:51.115714 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:51.451526 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:51.451488 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:51.813011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:51.809683 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:52.052974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:52.043103 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:52.171984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:52.166735 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:52.573012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:52.569390 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:52.604993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:52.603389 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:52.604993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:52.603459 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:53.522426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:53.522084 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:53.522426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:53.522142 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:53.569360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:53.569311 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:54.600585 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:54.600549 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:55.982046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:55.982015 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:56.076590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:56.076555 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:56.144601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:56.144567 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4"] Oct 30 18:26:56.144880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:56.144843 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" containerID="cri-o://5b7328890d765074b740d3cd0beb7616f1b3c2afa1f2c766f5fb6d74c256dbf7" gracePeriod=30 Oct 30 18:26:56.522761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:56.522722 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2"] Oct 30 18:26:56.522985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:56.522962 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" containerID="cri-o://774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b" gracePeriod=30 Oct 30 18:26:56.632342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:56.632313 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:57.706145 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:57.706111 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:57.809786 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:57.809754 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:58.746582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:58.746546 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:59.048793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:59.048691 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:59.151339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:59.151303 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:59.399735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:59.399702 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm"] Oct 30 18:26:59.400059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:59.400027 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" containerID="cri-o://7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e" gracePeriod=30 Oct 30 18:26:59.418704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:26:59.418675 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:00.868489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:00.868451 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:01.393982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.389046 5624 generic.go:356] "Generic (PLEG): container finished" podID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerID="5b7328890d765074b740d3cd0beb7616f1b3c2afa1f2c766f5fb6d74c256dbf7" exitCode=0 Oct 30 18:27:01.393982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.389127 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" event={"ID":"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6","Type":"ContainerDied","Data":"5b7328890d765074b740d3cd0beb7616f1b3c2afa1f2c766f5fb6d74c256dbf7"} Oct 30 18:27:01.455596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.455566 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:27:01.590408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.590383 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-token\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-token\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.590564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.590426 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-credential-socket\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.590564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.590471 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-socket\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.590564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.590501 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-envoy\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.590564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.590559 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-kube-api-access-hkcb9\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-kube-api-access-hkcb9\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.590769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.590631 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-data\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.590769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.590677 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istiod-ca-cert\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.590909 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.590888 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-certs\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.591265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.591236 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-credential-socket" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.591372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.591259 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-data" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.591372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.591263 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istiod-ca-cert" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:01.591491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.591466 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-certs" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.591582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.591561 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-podinfo\") pod \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\" (UID: \"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6\") " Oct 30 18:27:01.591841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.591818 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-socket" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.592613 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.592591 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.592752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.592739 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.592854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.592841 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.592961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.592948 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.593069 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.593054 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.593238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.593211 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-token" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:01.593517 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.593498 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-kube-api-access-hkcb9" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "kube-api-access-hkcb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:01.593993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.593906 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-podinfo" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:01.593993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.593991 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-envoy" pod "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" (UID: "a2ebbf0d-9cfe-452d-b066-5162a4ebfff6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.694793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.694716 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.694793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.694743 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.694793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.694752 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.694793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.694781 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcb9\" (UniqueName: \"kubernetes.io/projected/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6-kube-api-access-hkcb9\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.800124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.800097 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:27:01.897594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.897557 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-socket\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.898067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.897684 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-istio-token\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-istio-token\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.898067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.897831 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-envoy\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.898067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.897915 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-credential-socket\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.898067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.897916 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-socket" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.898067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.897992 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-kube-api-access-l8gjm\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-kube-api-access-l8gjm\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.898411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.898107 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-certs\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.898411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.898194 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f8db066-80e2-4003-919a-f92347b92fc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8db066-80e2-4003-919a-f92347b92fc4-istio-podinfo\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.898832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.898505 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-credential-socket" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.899131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.899105 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-certs" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.899243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.899177 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8db066-80e2-4003-919a-f92347b92fc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8db066-80e2-4003-919a-f92347b92fc4-istiod-ca-cert\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.899709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.899495 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8db066-80e2-4003-919a-f92347b92fc4-istiod-ca-cert" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:01.899709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.899533 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-data\") pod \"3f8db066-80e2-4003-919a-f92347b92fc4\" (UID: \"3f8db066-80e2-4003-919a-f92347b92fc4\") " Oct 30 18:27:01.899709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.899669 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-data" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.900514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.900490 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-envoy" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:01.900605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.900539 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.900605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.900560 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f8db066-80e2-4003-919a-f92347b92fc4-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.900605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.900574 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.900605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.900586 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.900605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.900598 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:01.901006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.900982 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-kube-api-access-l8gjm" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "kube-api-access-l8gjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:01.901794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.901773 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-istio-token" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:01.902625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:01.902602 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f8db066-80e2-4003-919a-f92347b92fc4-istio-podinfo" pod "3f8db066-80e2-4003-919a-f92347b92fc4" (UID: "3f8db066-80e2-4003-919a-f92347b92fc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:02.003110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.003012 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:02.003110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.003063 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f8db066-80e2-4003-919a-f92347b92fc4-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:02.003110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.003077 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gjm\" (UniqueName: \"kubernetes.io/projected/3f8db066-80e2-4003-919a-f92347b92fc4-kube-api-access-l8gjm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:02.003110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.003090 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f8db066-80e2-4003-919a-f92347b92fc4-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:02.401686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.401646 5624 generic.go:356] "Generic (PLEG): container finished" podID="3f8db066-80e2-4003-919a-f92347b92fc4" containerID="774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b" exitCode=0 Oct 30 18:27:02.401881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.401706 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" Oct 30 18:27:02.401881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.401729 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" event={"ID":"3f8db066-80e2-4003-919a-f92347b92fc4","Type":"ContainerDied","Data":"774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b"} Oct 30 18:27:02.401881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.401793 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2" event={"ID":"3f8db066-80e2-4003-919a-f92347b92fc4","Type":"ContainerDied","Data":"b1830a53c2b8014de37faf6ae16863bf4ad1e73da09ae201756454cd80022a3c"} Oct 30 18:27:02.401881 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.401823 5624 scope.go:122] "RemoveContainer" containerID="774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b" Oct 30 18:27:02.403559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.403520 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" Oct 30 18:27:02.403683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.403553 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4" event={"ID":"a2ebbf0d-9cfe-452d-b066-5162a4ebfff6","Type":"ContainerDied","Data":"a553f67c4f5fb97b86c0154036831bdedcd4ecad714fa1a51aa475436b09ab94"} Oct 30 18:27:02.415392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.415372 5624 scope.go:122] "RemoveContainer" containerID="774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b" Oct 30 18:27:02.415727 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:02.415699 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b\": container with ID starting with 774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b not found: ID does not exist" containerID="774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b" Oct 30 18:27:02.415819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.415735 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b"} err="failed to get container status \"774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b\": rpc error: code = NotFound desc = could not find container \"774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b\": container with ID starting with 774169e49d01a95dab69b94daa308a04a9377fc97aa5fbd8f6fe9d76ad5fe71b not found: ID does not exist" Oct 30 18:27:02.415819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.415760 5624 scope.go:122] "RemoveContainer" containerID="5b7328890d765074b740d3cd0beb7616f1b3c2afa1f2c766f5fb6d74c256dbf7" Oct 30 18:27:02.474205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.474166 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4"] Oct 30 18:27:02.487569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.487542 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-hddj4"] Oct 30 18:27:02.512822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.512795 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2"] Oct 30 18:27:02.535715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.535686 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7jkq2"] Oct 30 18:27:02.666203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:02.666124 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:03.793751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:03.793717 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" path="/var/lib/kubelet/pods/3f8db066-80e2-4003-919a-f92347b92fc4/volumes" Oct 30 18:27:03.794328 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:03.794312 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" path="/var/lib/kubelet/pods/a2ebbf0d-9cfe-452d-b066-5162a4ebfff6/volumes" Oct 30 18:27:04.593653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.593612 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:04.764312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.764286 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:27:04.894487 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894455 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-certs\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894510 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-data\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894713 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b51cc775-0d90-40a5-9c60-5d802467532a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51cc775-0d90-40a5-9c60-5d802467532a-istio-podinfo\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894762 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-envoy\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894763 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-certs" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894793 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-socket\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894840 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-kube-api-access-cbfjq\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-kube-api-access-cbfjq\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894774 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-data" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894902 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-istio-token\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-istio-token\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.894977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.894929 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-credential-socket\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.895445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895016 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51cc775-0d90-40a5-9c60-5d802467532a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51cc775-0d90-40a5-9c60-5d802467532a-istiod-ca-cert\") pod \"b51cc775-0d90-40a5-9c60-5d802467532a\" (UID: \"b51cc775-0d90-40a5-9c60-5d802467532a\") " Oct 30 18:27:04.895445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895052 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-socket" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:04.895445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895288 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-credential-socket" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:04.895629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895557 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51cc775-0d90-40a5-9c60-5d802467532a-istiod-ca-cert" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:04.895870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895848 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.895982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895875 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51cc775-0d90-40a5-9c60-5d802467532a-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.895982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895891 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.895982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895905 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.895982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.895918 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.897345 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.897320 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-istio-token" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:04.897460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.897357 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-kube-api-access-cbfjq" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "kube-api-access-cbfjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:04.897460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.897436 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-envoy" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:04.897460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.897441 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b51cc775-0d90-40a5-9c60-5d802467532a-istio-podinfo" pod "b51cc775-0d90-40a5-9c60-5d802467532a" (UID: "b51cc775-0d90-40a5-9c60-5d802467532a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:04.997642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.997564 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51cc775-0d90-40a5-9c60-5d802467532a-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.997642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.997589 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51cc775-0d90-40a5-9c60-5d802467532a-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.997642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.997599 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbfjq\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-kube-api-access-cbfjq\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.997642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:04.997608 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51cc775-0d90-40a5-9c60-5d802467532a-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:05.436761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.436723 5624 generic.go:356] "Generic (PLEG): container finished" podID="b51cc775-0d90-40a5-9c60-5d802467532a" containerID="7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e" exitCode=0 Oct 30 18:27:05.436919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.436777 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" Oct 30 18:27:05.436919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.436802 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" event={"ID":"b51cc775-0d90-40a5-9c60-5d802467532a","Type":"ContainerDied","Data":"7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e"} Oct 30 18:27:05.436919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.436863 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm" event={"ID":"b51cc775-0d90-40a5-9c60-5d802467532a","Type":"ContainerDied","Data":"b0837df51f1727871febbd818293c5232472e10cba135404fc7ffa83c054c478"} Oct 30 18:27:05.436919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.436888 5624 scope.go:122] "RemoveContainer" containerID="7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e" Oct 30 18:27:05.448772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.448754 5624 scope.go:122] "RemoveContainer" containerID="1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e" Oct 30 18:27:05.458775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.458754 5624 scope.go:122] "RemoveContainer" containerID="7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e" Oct 30 18:27:05.459096 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:05.459073 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e\": container with ID starting with 7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e not found: ID does not exist" containerID="7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e" Oct 30 18:27:05.459177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.459109 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e"} err="failed to get container status \"7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e\": rpc error: code = NotFound desc = could not find container \"7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e\": container with ID starting with 7ef423d07c628b5d57c3b62fa719099bba5beb8a5fa6f7c5b030f9608eb0fc8e not found: ID does not exist" Oct 30 18:27:05.459177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.459134 5624 scope.go:122] "RemoveContainer" containerID="1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e" Oct 30 18:27:05.459383 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:05.459365 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e\": container with ID starting with 1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e not found: ID does not exist" containerID="1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e" Oct 30 18:27:05.459421 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.459392 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e"} err="failed to get container status \"1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e\": rpc error: code = NotFound desc = could not find container \"1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e\": container with ID starting with 1922ed546cac7931cd2aef120c76b3bd4eb527627602e037b00af4771a605a1e not found: ID does not exist" Oct 30 18:27:05.508535 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.508509 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm"] Oct 30 18:27:05.521403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.521380 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-295497-openshift-default-58f5f7b8cc-wq5vm"] Oct 30 18:27:05.792366 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:05.792293 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" path="/var/lib/kubelet/pods/b51cc775-0d90-40a5-9c60-5d802467532a/volumes" Oct 30 18:27:06.112735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:06.112641 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:07.642837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:07.642804 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:09.226161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:09.226125 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:10.534131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:10.534097 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv"] Oct 30 18:27:10.571327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:10.571297 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786"] Oct 30 18:27:10.689226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:10.689181 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:10.802020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:10.800715 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:10.898286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:10.898245 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:12.628215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:12.628171 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:14.236206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:14.236177 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:15.470269 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:15.470228 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:15.537100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:15.537052 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" podUID="0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb" containerName="payload-processing" containerID="cri-o://162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1" gracePeriod=25 Oct 30 18:27:15.571788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:15.571743 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" podUID="4e202b77-f27a-4c86-84b0-ab8cee55c377" containerName="payload-pre-processing" containerID="cri-o://018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb" gracePeriod=25 Oct 30 18:27:15.908795 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:15.908750 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:16.060543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.060449 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:27:16.155991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.155966 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:27:16.249811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.249774 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e202b77-f27a-4c86-84b0-ab8cee55c377-kube-api-access-j5lhh\" (UniqueName: \"kubernetes.io/projected/4e202b77-f27a-4c86-84b0-ab8cee55c377-kube-api-access-j5lhh\") pod \"4e202b77-f27a-4c86-84b0-ab8cee55c377\" (UID: \"4e202b77-f27a-4c86-84b0-ab8cee55c377\") " Oct 30 18:27:16.250280 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.249862 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e202b77-f27a-4c86-84b0-ab8cee55c377-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e202b77-f27a-4c86-84b0-ab8cee55c377-plugins-config-volume\") pod \"4e202b77-f27a-4c86-84b0-ab8cee55c377\" (UID: \"4e202b77-f27a-4c86-84b0-ab8cee55c377\") " Oct 30 18:27:16.250280 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.249999 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-kube-api-access-j2r9g\" (UniqueName: \"kubernetes.io/projected/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-kube-api-access-j2r9g\") pod \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\" (UID: \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\") " Oct 30 18:27:16.250450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.250341 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e202b77-f27a-4c86-84b0-ab8cee55c377-plugins-config-volume" pod "4e202b77-f27a-4c86-84b0-ab8cee55c377" (UID: "4e202b77-f27a-4c86-84b0-ab8cee55c377"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:16.251047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.251020 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-plugins-config-volume\") pod \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\" (UID: \"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb\") " Oct 30 18:27:16.251579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.251547 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-plugins-config-volume" pod "0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb" (UID: "0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:16.252126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.252102 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e202b77-f27a-4c86-84b0-ab8cee55c377-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:16.252235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.252131 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:16.252642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.252617 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e202b77-f27a-4c86-84b0-ab8cee55c377-kube-api-access-j5lhh" pod "4e202b77-f27a-4c86-84b0-ab8cee55c377" (UID: "4e202b77-f27a-4c86-84b0-ab8cee55c377"). InnerVolumeSpecName "kube-api-access-j5lhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:16.253241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.253216 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-kube-api-access-j2r9g" pod "0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb" (UID: "0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb"). InnerVolumeSpecName "kube-api-access-j2r9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:16.353774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.353706 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5lhh\" (UniqueName: \"kubernetes.io/projected/4e202b77-f27a-4c86-84b0-ab8cee55c377-kube-api-access-j5lhh\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:16.353774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.353731 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2r9g\" (UniqueName: \"kubernetes.io/projected/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb-kube-api-access-j2r9g\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:16.563196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.563164 5624 generic.go:356] "Generic (PLEG): container finished" podID="0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb" containerID="162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1" exitCode=0 Oct 30 18:27:16.563658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.563226 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" Oct 30 18:27:16.563658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.563228 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" event={"ID":"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb","Type":"ContainerDied","Data":"162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1"} Oct 30 18:27:16.563658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.563313 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv" event={"ID":"0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb","Type":"ContainerDied","Data":"d77d44071c6f24dbe6a214fca253f508b026925ecc38ea8c14e925dd5616d357"} Oct 30 18:27:16.563658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.563398 5624 scope.go:122] "RemoveContainer" containerID="162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1" Oct 30 18:27:16.565527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.565503 5624 generic.go:356] "Generic (PLEG): container finished" podID="4e202b77-f27a-4c86-84b0-ab8cee55c377" containerID="018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb" exitCode=0 Oct 30 18:27:16.565643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.565537 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" event={"ID":"4e202b77-f27a-4c86-84b0-ab8cee55c377","Type":"ContainerDied","Data":"018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb"} Oct 30 18:27:16.565643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.565571 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" event={"ID":"4e202b77-f27a-4c86-84b0-ab8cee55c377","Type":"ContainerDied","Data":"eeab1290a8396e4b9868e91a1fd9e5c1ff72e748fea1e3605b2e7e028b6c22e5"} Oct 30 18:27:16.565643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.565576 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786" Oct 30 18:27:16.587651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.587632 5624 scope.go:122] "RemoveContainer" containerID="162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1" Oct 30 18:27:16.588056 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:16.588013 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1\": container with ID starting with 162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1 not found: ID does not exist" containerID="162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1" Oct 30 18:27:16.588156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.588067 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1"} err="failed to get container status \"162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1\": rpc error: code = NotFound desc = could not find container \"162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1\": container with ID starting with 162a9309903ba43ccda1a114a4e95cb7a5d21e165c823bbdbd75b6483cba14c1 not found: ID does not exist" Oct 30 18:27:16.588156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.588093 5624 scope.go:122] "RemoveContainer" containerID="018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb" Oct 30 18:27:16.598745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.598723 5624 scope.go:122] "RemoveContainer" containerID="018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb" Oct 30 18:27:16.599050 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:16.599030 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb\": container with ID starting with 018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb not found: ID does not exist" containerID="018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb" Oct 30 18:27:16.599153 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.599057 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb"} err="failed to get container status \"018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb\": rpc error: code = NotFound desc = could not find container \"018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb\": container with ID starting with 018e30eb965607b45daf267cfc90afdfa70cd651acf41e95dc30441758c7cdcb not found: ID does not exist" Oct 30 18:27:16.635969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.635927 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786"] Oct 30 18:27:16.648589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.648565 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-550276-849c97f58c-tj786"] Oct 30 18:27:16.683721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.683699 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv"] Oct 30 18:27:16.689645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.689621 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-550276-94746b766-wzfqv"] Oct 30 18:27:16.903678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:16.903605 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:17.298248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:17.298121 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:17.814824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:17.814789 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb" path="/var/lib/kubelet/pods/0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb/volumes" Oct 30 18:27:17.816310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:17.816286 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e202b77-f27a-4c86-84b0-ab8cee55c377" path="/var/lib/kubelet/pods/4e202b77-f27a-4c86-84b0-ab8cee55c377/volumes" Oct 30 18:27:17.964277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:17.964230 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:18.066921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:18.066834 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:18.169842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:18.169811 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:18.610713 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:18.610674 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:19.916425 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:19.916383 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:27:20.031461 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:20.031402 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:21.632590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:21.632555 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv"] Oct 30 18:27:21.633251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:21.633088 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" podUID="46237646-21b2-4830-9635-82d792e261c5" containerName="istio-proxy" containerID="cri-o://57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b" gracePeriod=30 Oct 30 18:27:21.647784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:21.647754 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj"] Oct 30 18:27:21.648028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:21.647983 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" containerID="cri-o://44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e" gracePeriod=30 Oct 30 18:27:21.657154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:21.657125 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:23.095408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.095366 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh"] Oct 30 18:27:23.096521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096497 5624 cpu_manager.go:414] "containerMap: removing container" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096522 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096539 5624 cpu_manager.go:414] "containerMap: removing container" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096547 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096559 5624 cpu_manager.go:414] "containerMap: removing container" podUID="ffa8cc09-b308-4348-af04-5773d5134665" containerName="maas-api" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096568 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa8cc09-b308-4348-af04-5773d5134665" containerName="maas-api" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096586 5624 cpu_manager.go:414] "containerMap: removing container" podUID="0ffdb840-fd18-4ad2-8635-2234ab2069b0" containerName="payload-pre-processing" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096594 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ffdb840-fd18-4ad2-8635-2234ab2069b0" containerName="payload-pre-processing" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096609 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" containerName="maas-api" Oct 30 18:27:23.096620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096616 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" containerName="maas-api" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096633 5624 cpu_manager.go:414] "containerMap: removing container" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096641 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096656 5624 cpu_manager.go:414] "containerMap: removing container" podUID="67687132-9d0b-4ebe-b344-3881f9369509" containerName="payload-processing" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096663 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67687132-9d0b-4ebe-b344-3881f9369509" containerName="payload-processing" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096677 5624 cpu_manager.go:414] "containerMap: removing container" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096685 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096699 5624 cpu_manager.go:414] "containerMap: removing container" podUID="13b40734-590f-4568-b812-c77303a52d56" containerName="payload-processing" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096706 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b40734-590f-4568-b812-c77303a52d56" containerName="payload-processing" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096715 5624 cpu_manager.go:414] "containerMap: removing container" podUID="0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb" containerName="payload-processing" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096722 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb" containerName="payload-processing" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096766 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4e202b77-f27a-4c86-84b0-ab8cee55c377" containerName="payload-pre-processing" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096774 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e202b77-f27a-4c86-84b0-ab8cee55c377" containerName="payload-pre-processing" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096815 5624 cpu_manager.go:414] "containerMap: removing container" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" Oct 30 18:27:23.097115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.096822 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097538 5624 cpu_manager.go:414] "containerMap: removing container" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097560 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097593 5624 cpu_manager.go:414] "containerMap: removing container" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097600 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097612 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097621 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097637 5624 cpu_manager.go:414] "containerMap: removing container" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097646 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097659 5624 cpu_manager.go:414] "containerMap: removing container" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097666 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097676 5624 cpu_manager.go:414] "containerMap: removing container" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097683 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097692 5624 cpu_manager.go:414] "containerMap: removing container" podUID="f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" containerName="payload-pre-processing" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097699 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" containerName="payload-pre-processing" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097716 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6899f054-fc85-459e-a1a7-9dee926b2c8c" containerName="payload-pre-processing" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097724 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6899f054-fc85-459e-a1a7-9dee926b2c8c" containerName="payload-pre-processing" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097736 5624 cpu_manager.go:414] "containerMap: removing container" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097743 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097766 5624 cpu_manager.go:414] "containerMap: removing container" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097774 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097807 5624 cpu_manager.go:414] "containerMap: removing container" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" Oct 30 18:27:23.097852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.097814 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098013 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8db066-80e2-4003-919a-f92347b92fc4" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098029 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098039 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098051 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="97247502-e6a3-44b3-aef5-cd9a3c71dd64" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098060 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ebbf0d-9cfe-452d-b066-5162a4ebfff6" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098072 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d633c4-c9c8-4b8e-948e-f44f73a6dbd2" containerName="payload-pre-processing" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098082 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="6899f054-fc85-459e-a1a7-9dee926b2c8c" containerName="payload-pre-processing" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098113 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10cb470-5e0a-4def-806d-0e08450f10a2" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098126 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a21199-1155-43cc-b36b-6215e6d3130b" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098142 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="aadedad1-fea9-471c-bd58-2372d6c91bb8" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098152 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098164 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa8cc09-b308-4348-af04-5773d5134665" containerName="maas-api" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098172 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ffdb840-fd18-4ad2-8635-2234ab2069b0" containerName="payload-pre-processing" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098181 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7a7f1d-20bd-41f1-b298-14543a6d2d1d" containerName="maas-api" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098192 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="1470d092-0818-4480-a80a-00b91adbeb4f" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098203 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="67687132-9d0b-4ebe-b344-3881f9369509" containerName="payload-processing" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098212 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51cc775-0d90-40a5-9c60-5d802467532a" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098220 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b40734-590f-4568-b812-c77303a52d56" containerName="payload-processing" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098228 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c71fc3c-dec9-43b2-806f-c9f8ebbe2abb" containerName="payload-processing" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098238 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e202b77-f27a-4c86-84b0-ab8cee55c377" containerName="payload-pre-processing" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098251 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" Oct 30 18:27:23.099055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.098263 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea5d15c-4957-49c0-9739-f5383e23f424" containerName="istio-proxy" Oct 30 18:27:23.104488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.104463 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.123412 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.123371 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0f0c50-serving-cert\"" Oct 30 18:27:23.123856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.123831 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh"] Oct 30 18:27:23.144580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.144512 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:27:23.144580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.144563 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" podUID="46237646-21b2-4830-9635-82d792e261c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:27:23.169644 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.169605 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:23.206743 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.206713 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fd0eb0-333a-431b-9395-1f2629097152-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.206888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.206860 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqdn\" (UniqueName: \"kubernetes.io/projected/49fd0eb0-333a-431b-9395-1f2629097152-kube-api-access-srqdn\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.206962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.206927 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.207083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.207065 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-metrics-certs\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.308326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.308284 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fd0eb0-333a-431b-9395-1f2629097152-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.308524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.308427 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqdn\" (UniqueName: \"kubernetes.io/projected/49fd0eb0-333a-431b-9395-1f2629097152-kube-api-access-srqdn\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.308524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.308501 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.308689 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.308591 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-metrics-certs\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.309119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.309089 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fd0eb0-333a-431b-9395-1f2629097152-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.309238 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:23.309206 5624 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-serving-cert: secret "maas-api-e2e-inf-a-0f0c50-serving-cert" not found Oct 30 18:27:23.309290 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:23.309268 5624 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls podName:49fd0eb0-333a-431b-9395-1f2629097152 nodeName:}" failed. No retries permitted until 2026-10-30 18:27:23.809251906 +0000 UTC m=+1538.197780001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls") pod "maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" (UID: "49fd0eb0-333a-431b-9395-1f2629097152") : secret "maas-api-e2e-inf-a-0f0c50-serving-cert" not found Oct 30 18:27:23.312711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.312683 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-metrics-certs\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.331605 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.331571 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqdn\" (UniqueName: \"kubernetes.io/projected/49fd0eb0-333a-431b-9395-1f2629097152-kube-api-access-srqdn\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.344175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.344140 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:23.445003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.444921 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:23.662073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.662043 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:23.768805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.768721 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:23.825099 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.825069 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.827919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.827889 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls\") pod \"maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:23.988838 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:23.988800 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:24.025632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:24.025545 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:24.640812 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:27:24.640783 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fd0eb0_333a_431b_9395_1f2629097152.slice/crio-16566956e1eb04972a6d1cec3c2168f6df9b087f39067b5c2dc78b7816163cd1 WatchSource:0}: Error finding container 16566956e1eb04972a6d1cec3c2168f6df9b087f39067b5c2dc78b7816163cd1: Status 404 returned error can't find the container with id 16566956e1eb04972a6d1cec3c2168f6df9b087f39067b5c2dc78b7816163cd1 Oct 30 18:27:24.641462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:24.641439 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh"] Oct 30 18:27:24.642240 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:24.642221 5624 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:27:24.673770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:24.673666 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" event={"ID":"49fd0eb0-333a-431b-9395-1f2629097152","Type":"ContainerStarted","Data":"16566956e1eb04972a6d1cec3c2168f6df9b087f39067b5c2dc78b7816163cd1"} Oct 30 18:27:24.794149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:24.794091 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:25.486929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:25.486844 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:25.580308 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:25.580273 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:25.667111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:25.667066 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:25.687197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:25.687156 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" event={"ID":"49fd0eb0-333a-431b-9395-1f2629097152","Type":"ContainerStarted","Data":"bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c"} Oct 30 18:27:25.688292 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:25.687876 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:25.736069 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:25.736018 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" podStartSLOduration=2.313286165 podStartE2EDuration="2.736000627s" totalImagesPullingTime="422.714462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:25.733243692 +0000 UTC m=+1540.121771809" watchObservedRunningTime="2026-10-30 18:27:25.736000627 +0000 UTC m=+1540.124528754" Oct 30 18:27:25.761374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:25.761307 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:26.114735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:26.114611 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:26.216406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:26.216345 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:26.962261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:26.962233 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:27:26.979482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:26.979462 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:27:27.080736 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.080702 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-certs\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.080887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.080774 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46237646-21b2-4830-9635-82d792e261c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46237646-21b2-4830-9635-82d792e261c5-istio-podinfo\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.080887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.080822 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-certs\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.080887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.080869 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-kube-api-access-mm8bq\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-kube-api-access-mm8bq\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.081133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.080901 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-data\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.081133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.080972 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-token\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-token\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.081133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081048 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-envoy\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.081133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081090 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-credential-socket\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.081133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081099 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-certs" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.081133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081130 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-kube-api-access-ffkpn\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-kube-api-access-ffkpn\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.081457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081222 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-socket\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.081457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081258 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-socket\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.081457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081287 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-istio-token\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-istio-token\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.081457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081359 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istiod-ca-cert\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.081457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081414 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-envoy\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.081457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081450 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46237646-21b2-4830-9635-82d792e261c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46237646-21b2-4830-9635-82d792e261c5-istiod-ca-cert\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.081790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081522 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-data\") pod \"46237646-21b2-4830-9635-82d792e261c5\" (UID: \"46237646-21b2-4830-9635-82d792e261c5\") " Oct 30 18:27:27.081790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081528 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-data" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.081790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081570 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-credential-socket\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.081790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081662 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-podinfo\") pod \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\" (UID: \"17022b7e-05e9-4f50-a89b-ac0dc7e3057b\") " Oct 30 18:27:27.081790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.081779 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-socket" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.082136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.082016 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-credential-socket" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.082946 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.082611 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.082946 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.082652 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.082946 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.082669 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.082946 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.082682 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.083232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.083008 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-credential-socket" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.083797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.083327 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46237646-21b2-4830-9635-82d792e261c5-istiod-ca-cert" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:27.084024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.083413 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-data" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.084024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.083500 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istiod-ca-cert" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:27.084024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.083568 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-envoy" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.084488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.084464 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-podinfo" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:27.085061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.085033 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-socket" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.085309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.085284 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-token" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:27.085794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.085769 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-kube-api-access-ffkpn" pod "17022b7e-05e9-4f50-a89b-ac0dc7e3057b" (UID: "17022b7e-05e9-4f50-a89b-ac0dc7e3057b"). InnerVolumeSpecName "kube-api-access-ffkpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:27.087903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.087878 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46237646-21b2-4830-9635-82d792e261c5-istio-podinfo" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:27.088387 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.088363 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-kube-api-access-mm8bq" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "kube-api-access-mm8bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:27.088482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.088395 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-istio-token" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:27.090057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.090024 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-envoy" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.091859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.091834 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-certs" pod "46237646-21b2-4830-9635-82d792e261c5" (UID: "46237646-21b2-4830-9635-82d792e261c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184350 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffkpn\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-kube-api-access-ffkpn\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184376 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184387 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184395 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184403 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184411 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46237646-21b2-4830-9635-82d792e261c5-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184418 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184426 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184433 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.184432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184442 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46237646-21b2-4830-9635-82d792e261c5-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.185071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184450 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46237646-21b2-4830-9635-82d792e261c5-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.185071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184457 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm8bq\" (UniqueName: \"kubernetes.io/projected/46237646-21b2-4830-9635-82d792e261c5-kube-api-access-mm8bq\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.185071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184465 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.185071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.184474 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17022b7e-05e9-4f50-a89b-ac0dc7e3057b-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:27.642870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.642801 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:27.736993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.734722 5624 generic.go:356] "Generic (PLEG): container finished" podID="46237646-21b2-4830-9635-82d792e261c5" containerID="57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b" exitCode=0 Oct 30 18:27:27.736993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.734872 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" event={"ID":"46237646-21b2-4830-9635-82d792e261c5","Type":"ContainerDied","Data":"57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b"} Oct 30 18:27:27.736993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.734946 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" event={"ID":"46237646-21b2-4830-9635-82d792e261c5","Type":"ContainerDied","Data":"72e4ab473b80e22ac571663a158d906fa1d2a169b160abe10581cdc4c097511f"} Oct 30 18:27:27.736993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.734986 5624 scope.go:122] "RemoveContainer" containerID="57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b" Oct 30 18:27:27.736993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.735212 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv" Oct 30 18:27:27.739503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.739477 5624 generic.go:356] "Generic (PLEG): container finished" podID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerID="44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e" exitCode=0 Oct 30 18:27:27.739864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.739611 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" Oct 30 18:27:27.740315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.739633 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" event={"ID":"17022b7e-05e9-4f50-a89b-ac0dc7e3057b","Type":"ContainerDied","Data":"44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e"} Oct 30 18:27:27.740315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.740275 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj" event={"ID":"17022b7e-05e9-4f50-a89b-ac0dc7e3057b","Type":"ContainerDied","Data":"b81c76a7e3ae7433199ac5b763c610f58cd5f2a1de2d079cfb2a6d97a1d40b66"} Oct 30 18:27:27.777295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.777131 5624 scope.go:122] "RemoveContainer" containerID="57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b" Oct 30 18:27:27.777575 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:27.777546 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b\": container with ID starting with 57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b not found: ID does not exist" containerID="57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b" Oct 30 18:27:27.777674 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.777583 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b"} err="failed to get container status \"57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b\": rpc error: code = NotFound desc = could not find container \"57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b\": container with ID starting with 57b39b135783af836fb3651ccc7303f3a99f2e7d36e12b6c7bd15649a110dc9b not found: ID does not exist" Oct 30 18:27:27.777674 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.777602 5624 scope.go:122] "RemoveContainer" containerID="44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e" Oct 30 18:27:27.801094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.801067 5624 scope.go:122] "RemoveContainer" containerID="44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e" Oct 30 18:27:27.801743 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:27.801501 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e\": container with ID starting with 44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e not found: ID does not exist" containerID="44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e" Oct 30 18:27:27.801875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.801760 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e"} err="failed to get container status \"44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e\": rpc error: code = NotFound desc = could not find container \"44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e\": container with ID starting with 44fab34473e5fc4f26bdab5ac49e1bf7907b236d8441dbea4468c8171117834e not found: ID does not exist" Oct 30 18:27:27.814775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.814752 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv"] Oct 30 18:27:27.828489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.828467 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-z4kxv"] Oct 30 18:27:27.865289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.865255 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj"] Oct 30 18:27:27.870148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.870124 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-qv6jj"] Oct 30 18:27:27.908689 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.908606 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn"] Oct 30 18:27:27.908869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:27.908843 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" containerID="cri-o://6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930" gracePeriod=30 Oct 30 18:27:28.300350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:28.300067 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:27:29.635086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:29.635048 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:29.793047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:29.793016 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" path="/var/lib/kubelet/pods/17022b7e-05e9-4f50-a89b-ac0dc7e3057b/volumes" Oct 30 18:27:29.793556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:29.793541 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46237646-21b2-4830-9635-82d792e261c5" path="/var/lib/kubelet/pods/46237646-21b2-4830-9635-82d792e261c5/volumes" Oct 30 18:27:31.351823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:31.351793 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:31.704760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:31.704727 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:27:32.349700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:32.349660 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:32.450383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:32.450334 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:32.674372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:32.674339 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:33.202917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.202851 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:27:33.236392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236359 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istiod-ca-cert\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.236554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236493 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-certs\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.236554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236541 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-podinfo\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.236671 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236591 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-credential-socket\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.236671 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236624 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-kube-api-access-clv2l\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-kube-api-access-clv2l\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.236802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236780 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-certs" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:33.236872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236798 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istiod-ca-cert" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:33.236872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236823 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-data\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.236872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.236868 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-socket\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.237066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237005 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-envoy\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.237066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237035 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-credential-socket" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:33.237178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237061 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-data" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:33.237178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237068 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-token\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-token\") pod \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\" (UID: \"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0\") " Oct 30 18:27:33.237178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237164 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-socket" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:33.237771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237750 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.237771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237771 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.237990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237780 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.237990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237791 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.237990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.237798 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.239084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.239061 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-kube-api-access-clv2l" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "kube-api-access-clv2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:33.239198 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.239179 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-envoy" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:33.239381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.239355 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-token" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:33.239446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.239401 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-podinfo" pod "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" (UID: "1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:33.269514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.269482 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:33.339724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.339694 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.339724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.339720 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clv2l\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-kube-api-access-clv2l\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.339724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.339730 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.339977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.339740 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:33.885856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.885815 5624 generic.go:356] "Generic (PLEG): container finished" podID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerID="6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930" exitCode=0 Oct 30 18:27:33.886348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.885878 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" event={"ID":"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0","Type":"ContainerDied","Data":"6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930"} Oct 30 18:27:33.886348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.885909 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" Oct 30 18:27:33.886348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.885951 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn" event={"ID":"1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0","Type":"ContainerDied","Data":"94bce3a27673e73f1c24ea81869cd96c758bd0fbd8de2940d2364fa2980efd2a"} Oct 30 18:27:33.886348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.885981 5624 scope.go:122] "RemoveContainer" containerID="6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930" Oct 30 18:27:33.902499 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.902151 5624 scope.go:122] "RemoveContainer" containerID="6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930" Oct 30 18:27:33.903164 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:33.903098 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930\": container with ID starting with 6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930 not found: ID does not exist" containerID="6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930" Oct 30 18:27:33.903164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.903142 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930"} err="failed to get container status \"6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930\": rpc error: code = NotFound desc = could not find container \"6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930\": container with ID starting with 6503e9b9ee3d59a8d91c0087b97132a77855598784d14190715a1cf1e2db2930 not found: ID does not exist" Oct 30 18:27:33.959306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.959277 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn"] Oct 30 18:27:33.969742 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:33.969718 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8z5rn"] Oct 30 18:27:34.888694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:34.888654 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:35.792360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:35.792327 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" path="/var/lib/kubelet/pods/1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0/volumes" Oct 30 18:27:36.658524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:36.658490 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:37.254427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:37.254390 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb"] Oct 30 18:27:37.288976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:37.288929 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4"] Oct 30 18:27:37.410275 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:37.410236 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:37.541458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:37.541363 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:38.439704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:38.439671 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:39.928204 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:39.928165 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:41.448232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:41.448196 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:42.228336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.228299 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:42.255375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.255322 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" podUID="b1ea3dea-cf99-46ba-8ae0-7e827da688e8" containerName="payload-processing" containerID="cri-o://28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360" gracePeriod=25 Oct 30 18:27:42.290182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.290153 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" podUID="2c427900-f342-4cff-878b-4624a9165ae5" containerName="payload-pre-processing" containerID="cri-o://057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72" gracePeriod=25 Oct 30 18:27:42.586458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.586431 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:27:42.647347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.647326 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:27:42.700715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.700685 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-kube-api-access-wmgdp\" (UniqueName: \"kubernetes.io/projected/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-kube-api-access-wmgdp\") pod \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\" (UID: \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\") " Oct 30 18:27:42.700973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.700944 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-plugins-config-volume\") pod \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\" (UID: \"b1ea3dea-cf99-46ba-8ae0-7e827da688e8\") " Oct 30 18:27:42.701267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.701242 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-plugins-config-volume" pod "b1ea3dea-cf99-46ba-8ae0-7e827da688e8" (UID: "b1ea3dea-cf99-46ba-8ae0-7e827da688e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:42.701629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.701608 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.703780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.703758 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-kube-api-access-wmgdp" pod "b1ea3dea-cf99-46ba-8ae0-7e827da688e8" (UID: "b1ea3dea-cf99-46ba-8ae0-7e827da688e8"). InnerVolumeSpecName "kube-api-access-wmgdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:42.802779 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.802702 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c427900-f342-4cff-878b-4624a9165ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c427900-f342-4cff-878b-4624a9165ae5-plugins-config-volume\") pod \"2c427900-f342-4cff-878b-4624a9165ae5\" (UID: \"2c427900-f342-4cff-878b-4624a9165ae5\") " Oct 30 18:27:42.802779 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.802763 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c427900-f342-4cff-878b-4624a9165ae5-kube-api-access-lg7c8\" (UniqueName: \"kubernetes.io/projected/2c427900-f342-4cff-878b-4624a9165ae5-kube-api-access-lg7c8\") pod \"2c427900-f342-4cff-878b-4624a9165ae5\" (UID: \"2c427900-f342-4cff-878b-4624a9165ae5\") " Oct 30 18:27:42.803079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.803058 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c427900-f342-4cff-878b-4624a9165ae5-plugins-config-volume" pod "2c427900-f342-4cff-878b-4624a9165ae5" (UID: "2c427900-f342-4cff-878b-4624a9165ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:42.803355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.803332 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c427900-f342-4cff-878b-4624a9165ae5-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.803355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.803354 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmgdp\" (UniqueName: \"kubernetes.io/projected/b1ea3dea-cf99-46ba-8ae0-7e827da688e8-kube-api-access-wmgdp\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.804808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.804791 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c427900-f342-4cff-878b-4624a9165ae5-kube-api-access-lg7c8" pod "2c427900-f342-4cff-878b-4624a9165ae5" (UID: "2c427900-f342-4cff-878b-4624a9165ae5"). InnerVolumeSpecName "kube-api-access-lg7c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:42.905422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.905386 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg7c8\" (UniqueName: \"kubernetes.io/projected/2c427900-f342-4cff-878b-4624a9165ae5-kube-api-access-lg7c8\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:42.976410 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.976372 5624 generic.go:356] "Generic (PLEG): container finished" podID="2c427900-f342-4cff-878b-4624a9165ae5" containerID="057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72" exitCode=0 Oct 30 18:27:42.976586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.976419 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" Oct 30 18:27:42.976586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.976457 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" event={"ID":"2c427900-f342-4cff-878b-4624a9165ae5","Type":"ContainerDied","Data":"057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72"} Oct 30 18:27:42.976586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.976512 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4" event={"ID":"2c427900-f342-4cff-878b-4624a9165ae5","Type":"ContainerDied","Data":"7dd7be866e7ecd80a5eac9df4acb81d6626392fe3fed6cf540c8a0f3c305659c"} Oct 30 18:27:42.976586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.976542 5624 scope.go:122] "RemoveContainer" containerID="057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72" Oct 30 18:27:42.978049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.978029 5624 generic.go:356] "Generic (PLEG): container finished" podID="b1ea3dea-cf99-46ba-8ae0-7e827da688e8" containerID="28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360" exitCode=0 Oct 30 18:27:42.978166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.978079 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" Oct 30 18:27:42.978166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.978070 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" event={"ID":"b1ea3dea-cf99-46ba-8ae0-7e827da688e8","Type":"ContainerDied","Data":"28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360"} Oct 30 18:27:42.978289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.978196 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb" event={"ID":"b1ea3dea-cf99-46ba-8ae0-7e827da688e8","Type":"ContainerDied","Data":"8e6a67778f022672ee77a5f385460a92be8de57993425523a03b37e7e445941e"} Oct 30 18:27:42.989888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.989871 5624 scope.go:122] "RemoveContainer" containerID="057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72" Oct 30 18:27:42.990170 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:42.990154 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72\": container with ID starting with 057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72 not found: ID does not exist" containerID="057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72" Oct 30 18:27:42.990218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.990177 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72"} err="failed to get container status \"057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72\": rpc error: code = NotFound desc = could not find container \"057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72\": container with ID starting with 057c71c938ff6e0bd28c26636e1d6a2ee18e4b1b8d958718455ef76aa9335f72 not found: ID does not exist" Oct 30 18:27:42.990218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.990198 5624 scope.go:122] "RemoveContainer" containerID="28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360" Oct 30 18:27:42.999384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.999368 5624 scope.go:122] "RemoveContainer" containerID="28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360" Oct 30 18:27:42.999616 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:42.999598 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360\": container with ID starting with 28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360 not found: ID does not exist" containerID="28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360" Oct 30 18:27:42.999671 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:42.999620 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360"} err="failed to get container status \"28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360\": rpc error: code = NotFound desc = could not find container \"28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360\": container with ID starting with 28e51576c0a764e1a767bdaf6d07bc437f50e505cf9c7259322e826b0f5a6360 not found: ID does not exist" Oct 30 18:27:43.032856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.032833 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4"] Oct 30 18:27:43.045447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.045426 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-227c01-6b9459586c-2r6w4"] Oct 30 18:27:43.083489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.082587 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb"] Oct 30 18:27:43.089204 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.089184 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-227c01-cb946789d-9nbjb"] Oct 30 18:27:43.130429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.130405 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:43.792973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.792919 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c427900-f342-4cff-878b-4624a9165ae5" path="/var/lib/kubelet/pods/2c427900-f342-4cff-878b-4624a9165ae5/volumes" Oct 30 18:27:43.793592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.793570 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ea3dea-cf99-46ba-8ae0-7e827da688e8" path="/var/lib/kubelet/pods/b1ea3dea-cf99-46ba-8ae0-7e827da688e8/volumes" Oct 30 18:27:43.882225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.882191 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:43.993967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:43.987225 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:44.952540 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:44.952502 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:46.875263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:46.875232 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:48.754316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:48.754281 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:49.939595 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:49.939553 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:27:50.922319 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:50.916662 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.032824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.032786 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.307544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.307454 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm"] Oct 30 18:27:52.308118 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.307729 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" podUID="41a22e04-5ee6-4bd4-8848-6f8f96a65eac" containerName="maas-api" containerID="cri-o://49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6" gracePeriod=30 Oct 30 18:27:52.384215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.383564 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.493635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.493599 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.912022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.880649 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.926141 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn"] Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927253 5624 cpu_manager.go:414] "containerMap: removing container" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927274 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927288 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927296 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927324 5624 cpu_manager.go:414] "containerMap: removing container" podUID="b1ea3dea-cf99-46ba-8ae0-7e827da688e8" containerName="payload-processing" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927332 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ea3dea-cf99-46ba-8ae0-7e827da688e8" containerName="payload-processing" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927350 5624 cpu_manager.go:414] "containerMap: removing container" podUID="46237646-21b2-4830-9635-82d792e261c5" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927357 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46237646-21b2-4830-9635-82d792e261c5" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927369 5624 cpu_manager.go:414] "containerMap: removing container" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927375 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927423 5624 cpu_manager.go:414] "containerMap: removing container" podUID="2c427900-f342-4cff-878b-4624a9165ae5" containerName="payload-pre-processing" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927430 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c427900-f342-4cff-878b-4624a9165ae5" containerName="payload-pre-processing" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927472 5624 cpu_manager.go:414] "containerMap: removing container" podUID="41a22e04-5ee6-4bd4-8848-6f8f96a65eac" containerName="maas-api" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927479 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a22e04-5ee6-4bd4-8848-6f8f96a65eac" containerName="maas-api" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927652 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c427900-f342-4cff-878b-4624a9165ae5" containerName="payload-pre-processing" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927678 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a22e04-5ee6-4bd4-8848-6f8f96a65eac" containerName="maas-api" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927694 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="1773b5ea-e573-4ae6-a5d1-ea5e02a7c0b0" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927703 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4d072e-56c8-4f49-966c-18445c91eea2" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927712 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac309921-c2c5-4c6a-8635-79dea80cfadf" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927722 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ea3dea-cf99-46ba-8ae0-7e827da688e8" containerName="payload-processing" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927731 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="46237646-21b2-4830-9635-82d792e261c5" containerName="istio-proxy" Oct 30 18:27:52.930057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.927740 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="17022b7e-05e9-4f50-a89b-ac0dc7e3057b" containerName="istio-proxy" Oct 30 18:27:52.964344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.964220 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-metrics-certs\") pod \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " Oct 30 18:27:52.964506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.964395 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-maas-api-tls\") pod \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " Oct 30 18:27:52.964571 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.964552 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-openshift-service-ca\") pod \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " Oct 30 18:27:52.964625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.964604 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-kube-api-access-qbh9r\" (UniqueName: \"kubernetes.io/projected/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-kube-api-access-qbh9r\") pod \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\" (UID: \"41a22e04-5ee6-4bd4-8848-6f8f96a65eac\") " Oct 30 18:27:52.965990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.964813 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:52.969976 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.968965 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-maas-api-tls" pod "41a22e04-5ee6-4bd4-8848-6f8f96a65eac" (UID: "41a22e04-5ee6-4bd4-8848-6f8f96a65eac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:52.971081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.971044 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-kube-api-access-qbh9r" pod "41a22e04-5ee6-4bd4-8848-6f8f96a65eac" (UID: "41a22e04-5ee6-4bd4-8848-6f8f96a65eac"). InnerVolumeSpecName "kube-api-access-qbh9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:52.971249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.971222 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-openshift-service-ca" pod "41a22e04-5ee6-4bd4-8848-6f8f96a65eac" (UID: "41a22e04-5ee6-4bd4-8848-6f8f96a65eac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:52.972260 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.972235 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-metrics-certs" pod "41a22e04-5ee6-4bd4-8848-6f8f96a65eac" (UID: "41a22e04-5ee6-4bd4-8848-6f8f96a65eac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:52.983288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.983235 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-69c44c-openshift-default-dockercfg-b67cf\"" Oct 30 18:27:52.988218 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.988088 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.990995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:52.990949 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn"] Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.123426 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-podinfo\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.123584 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-certs\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.123623 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-socket\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.123681 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9bf78e-2113-4935-a722-92de3a6be1b1-istiod-ca-cert\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.123760 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-token\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.123816 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-credential-socket\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.123976 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-data\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.124058 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-envoy\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.124122 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hszsh\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-kube-api-access-hszsh\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.124281 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.124298 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.124314 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbh9r\" (UniqueName: \"kubernetes.io/projected/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-kube-api-access-qbh9r\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.124328 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a22e04-5ee6-4bd4-8848-6f8f96a65eac-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:27:53.131962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.130395 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:53.205974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.201145 5624 generic.go:356] "Generic (PLEG): container finished" podID="41a22e04-5ee6-4bd4-8848-6f8f96a65eac" containerID="49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6" exitCode=0 Oct 30 18:27:53.205974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.201248 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" event={"ID":"41a22e04-5ee6-4bd4-8848-6f8f96a65eac","Type":"ContainerDied","Data":"49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6"} Oct 30 18:27:53.205974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.201299 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" event={"ID":"41a22e04-5ee6-4bd4-8848-6f8f96a65eac","Type":"ContainerDied","Data":"d7add461e1c69a07cb60eca4250651f871071d782fece8eaab27b405337579a3"} Oct 30 18:27:53.205974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.201328 5624 scope.go:122] "RemoveContainer" containerID="49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6" Oct 30 18:27:53.205974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.201494 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227007 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-podinfo\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227136 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-certs\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227165 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-socket\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227208 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9bf78e-2113-4935-a722-92de3a6be1b1-istiod-ca-cert\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227271 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-token\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227362 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-credential-socket\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227481 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-data\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227547 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-envoy\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.227608 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hszsh\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-kube-api-access-hszsh\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.228317 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9bf78e-2113-4935-a722-92de3a6be1b1-istiod-ca-cert\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.228482 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-credential-socket\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.228555 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-certs\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.228712 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-data\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.229049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.228794 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-socket\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.230165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.230072 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:53.237856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.237762 5624 scope.go:122] "RemoveContainer" containerID="49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6" Oct 30 18:27:53.238427 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:27:53.238322 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6\": container with ID starting with 49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6 not found: ID does not exist" containerID="49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6" Oct 30 18:27:53.238427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.238361 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6"} err="failed to get container status \"49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6\": rpc error: code = NotFound desc = could not find container \"49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6\": container with ID starting with 49b76ca4d6561f7e1134d5a34777119784c09ac3b95dadabfd1a211f36c8d9d6 not found: ID does not exist" Oct 30 18:27:53.241756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.241735 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-podinfo\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.243157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.243137 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-envoy\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.285703 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.283286 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-token\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.291005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.287630 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hszsh\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-kube-api-access-hszsh\") pod \"e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.292460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.292405 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm"] Oct 30 18:27:53.298023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.298000 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-550276-678dff9d59-9snzm"] Oct 30 18:27:53.313243 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.310923 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:53.331628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.331534 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:53.708011 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.707982 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:53.713852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.713754 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn"] Oct 30 18:27:53.717767 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:27:53.717730 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9bf78e_2113_4935_a722_92de3a6be1b1.slice/crio-92eeb9031e6942d25238dc493c46c425cb3e8f68bfc27724b63abab3343f3751 WatchSource:0}: Error finding container 92eeb9031e6942d25238dc493c46c425cb3e8f68bfc27724b63abab3343f3751: Status 404 returned error can't find the container with id 92eeb9031e6942d25238dc493c46c425cb3e8f68bfc27724b63abab3343f3751 Oct 30 18:27:53.729307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.729265 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:27:53.729386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.729355 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:27:53.729432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.729398 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:27:53.797643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.797611 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a22e04-5ee6-4bd4-8848-6f8f96a65eac" path="/var/lib/kubelet/pods/41a22e04-5ee6-4bd4-8848-6f8f96a65eac/volumes" Oct 30 18:27:53.910370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:53.910339 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:54.213773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:54.213732 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" event={"ID":"ac9bf78e-2113-4935-a722-92de3a6be1b1","Type":"ContainerStarted","Data":"a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4"} Oct 30 18:27:54.214325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:54.213797 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" event={"ID":"ac9bf78e-2113-4935-a722-92de3a6be1b1","Type":"ContainerStarted","Data":"92eeb9031e6942d25238dc493c46c425cb3e8f68bfc27724b63abab3343f3751"} Oct 30 18:27:54.266573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:54.266515 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" podStartSLOduration=2.266495682 podStartE2EDuration="2.266495682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:54.258021395 +0000 UTC m=+1568.646549513" watchObservedRunningTime="2026-10-30 18:27:54.266495682 +0000 UTC m=+1568.655023815" Oct 30 18:27:54.312476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:54.312436 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:54.319265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:54.319229 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:55.162802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:55.162770 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:55.269389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:55.267345 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:55.274174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:55.274145 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:27:56.272802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.272763 5624 generic.go:356] "Generic (PLEG): container finished" podID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerID="a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6" exitCode=0 Oct 30 18:27:56.273357 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.272832 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" event={"ID":"f5b4e3bb-6754-4431-bd4a-c2b1512457ba","Type":"ContainerDied","Data":"a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6"} Oct 30 18:27:56.273926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.273900 5624 scope.go:122] "RemoveContainer" containerID="a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6" Oct 30 18:27:56.276637 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.276603 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:27:56.276752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.276693 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:27:56.276752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.276735 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:27:56.680959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.679989 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:56.793958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.793899 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:56.913458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:56.913420 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:57.440688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:57.417728 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:57.646115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:57.644640 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" event={"ID":"f5b4e3bb-6754-4431-bd4a-c2b1512457ba","Type":"ContainerStarted","Data":"7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a"} Oct 30 18:27:57.646115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:57.645544 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:27:57.790340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:57.787145 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:27:57.790340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:57.787213 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:27:58.748111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:58.748075 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:27:58.748737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:58.748687 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:27:59.092021 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.080761 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.093160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.092422 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.125626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.125577 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.155073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.155021 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.155567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.155287 5624 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Oct 30 18:27:59.297960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.288535 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:59.660974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.635640 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:59.702769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.702735 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:27:59.723154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:27:59.723114 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:28:00.087146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.087049 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 18:28:00.088892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.088862 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Oct 30 18:28:00.089907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.089887 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:28:00.117957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.095290 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:00.126990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.126275 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 30 18:28:00.204128 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.204102 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:28:00.214103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.214080 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:28:00.406533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.406505 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:28:00.481982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.455438 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:28:00.582967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.582912 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:28:00.586108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.584957 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:00.640368 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.640327 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:28:00.640557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:00.640390 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:28:01.175992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:01.175748 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:01.305863 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:01.305828 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:01.545971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:01.545869 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:01.642522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:01.642476 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:28:01.642865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:01.642832 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:28:01.678483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:01.678402 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:02.025369 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:02.025332 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:02.640183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:02.640147 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:28:02.640575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:02.640208 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:28:03.637662 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:03.637633 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:03.639761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:03.639737 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 18:28:03.639828 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:03.639790 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 18:28:03.821623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:03.821589 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:28:04.644408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:04.644370 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:28:04.645654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:04.645633 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:28:05.881452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:05.881422 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:06.777096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.777066 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf"] Oct 30 18:28:06.782076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.782055 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.838726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.838695 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf"] Oct 30 18:28:06.852063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.852031 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd"] Oct 30 18:28:06.857659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.857622 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd"] Oct 30 18:28:06.857878 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.857835 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.879287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879261 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.879417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879322 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f87e377-56d3-4931-8d5e-e9d479166009-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.879503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879442 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.879503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879495 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879602 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879635 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879656 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879691 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879742 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f87e377-56d3-4931-8d5e-e9d479166009-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879831 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879850 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879866 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879896 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.879911 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccmwp\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-kube-api-access-ccmwp\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.880042 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.880066 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.880095 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.880134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.880112 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxdj\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-kube-api-access-ggxdj\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.981752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981724 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981773 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f87e377-56d3-4931-8d5e-e9d479166009-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981809 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981831 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981864 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981885 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981905 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981971 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.981996 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f87e377-56d3-4931-8d5e-e9d479166009-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982054 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982075 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982103 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982146 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982171 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccmwp\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-kube-api-access-ccmwp\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982206 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982220 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.982239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982243 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.983159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982290 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.983159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982316 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxdj\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-kube-api-access-ggxdj\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.983159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982359 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.983159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982425 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.983159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982675 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.983159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982728 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.983159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982913 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.983159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.982916 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.983609 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.983286 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.983609 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.983351 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.983609 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.983600 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f87e377-56d3-4931-8d5e-e9d479166009-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.984650 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.984605 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:06.984778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.984758 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.985049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.985031 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f87e377-56d3-4931-8d5e-e9d479166009-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:06.985582 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:06.985559 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:07.012035 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.011981 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccmwp\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-kube-api-access-ccmwp\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:07.012370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.012330 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxdj\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-kube-api-access-ggxdj\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:07.012483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.012462 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:07.012904 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.012881 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:07.033235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.033175 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2"] Oct 30 18:28:07.098391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.098329 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:07.102020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.101992 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:07.178065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.177721 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:07.425115 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:28:07.425083 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dcb1f4_4b8e_44c0_a306_a94cafcca635.slice/crio-fe1f6cbba58b6633da57a2b60feff6043e6ab5c6671f1b32ae135b26c2d5acc5 WatchSource:0}: Error finding container fe1f6cbba58b6633da57a2b60feff6043e6ab5c6671f1b32ae135b26c2d5acc5: Status 404 returned error can't find the container with id fe1f6cbba58b6633da57a2b60feff6043e6ab5c6671f1b32ae135b26c2d5acc5 Oct 30 18:28:07.428628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.428589 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:07.428721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.428679 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:07.428761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.428721 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:07.429749 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.429723 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf"] Oct 30 18:28:07.553347 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:28:07.553320 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f87e377_56d3_4931_8d5e_e9d479166009.slice/crio-0360978c4132d15a8c33c1b737614213cb85f39673d7d65de70b7802a5060db2 WatchSource:0}: Error finding container 0360978c4132d15a8c33c1b737614213cb85f39673d7d65de70b7802a5060db2: Status 404 returned error can't find the container with id 0360978c4132d15a8c33c1b737614213cb85f39673d7d65de70b7802a5060db2 Oct 30 18:28:07.555884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.555859 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd"] Oct 30 18:28:07.556363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.556332 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:07.556447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.556399 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:07.556447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.556433 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:07.747970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:07.747905 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:08.267711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:08.267670 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" event={"ID":"97dcb1f4-4b8e-44c0-a306-a94cafcca635","Type":"ContainerStarted","Data":"4be7100ade67cf05006cba0b406a6647d3f27abebb57bb627d449e327748187c"} Oct 30 18:28:08.268282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:08.267731 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" event={"ID":"97dcb1f4-4b8e-44c0-a306-a94cafcca635","Type":"ContainerStarted","Data":"fe1f6cbba58b6633da57a2b60feff6043e6ab5c6671f1b32ae135b26c2d5acc5"} Oct 30 18:28:08.269484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:08.269458 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" event={"ID":"7f87e377-56d3-4931-8d5e-e9d479166009","Type":"ContainerStarted","Data":"0e8103da125a531de6c8ae8ef523c201a48abfd1b72f86d9d78a3417adc2c474"} Oct 30 18:28:08.269586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:08.269500 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" event={"ID":"7f87e377-56d3-4931-8d5e-e9d479166009","Type":"ContainerStarted","Data":"0360978c4132d15a8c33c1b737614213cb85f39673d7d65de70b7802a5060db2"} Oct 30 18:28:08.317088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:08.317027 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" podStartSLOduration=2.317008867 podStartE2EDuration="2.317008867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:08.309344755 +0000 UTC m=+1582.697872870" watchObservedRunningTime="2026-10-30 18:28:08.317008867 +0000 UTC m=+1582.705536984" Oct 30 18:28:08.364465 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:08.364388 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" podStartSLOduration=2.36436801 podStartE2EDuration="2.36436801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:08.357237216 +0000 UTC m=+1582.745765332" watchObservedRunningTime="2026-10-30 18:28:08.36436801 +0000 UTC m=+1582.752896127" Oct 30 18:28:09.106958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:09.099425 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:09.127965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:09.120074 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:09.127965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:09.120154 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 18:28:09.178982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:09.178925 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:09.180440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:09.180419 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:28:09.181092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:09.180459 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 18:28:09.457299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:09.457266 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:10.104960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:10.101271 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:10.104960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:10.101338 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 18:28:10.178946 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:10.178900 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:28:10.179115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:10.178983 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 18:28:10.863208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:10.862988 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:11.122498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:11.122463 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:11.122830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:11.122799 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 18:28:11.201371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:11.201334 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:28:11.201654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:11.201625 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 18:28:11.671964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:11.669764 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:11.838961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:11.818063 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:12.049245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:12.049146 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:12.104957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:12.101282 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:12.104957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:12.101354 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 18:28:12.183403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:12.183370 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:28:12.183684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:12.183653 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 18:28:12.351300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:12.351185 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:13.101306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:13.101272 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:13.101903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:13.101875 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 18:28:13.155128 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:13.155092 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:13.178983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:13.178918 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:28:13.179220 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:13.179004 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 18:28:13.262088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:13.261621 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:14.103692 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:14.103617 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 18:28:14.104305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:14.104274 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 18:28:14.182959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:14.182314 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 18:28:14.182959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:14.182397 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 18:28:14.394963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:14.391455 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:15.105248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:15.105219 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:15.105967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:15.105921 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:15.106547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:15.106527 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:15.183559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:15.183534 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:15.183822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:15.183799 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:15.184447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:15.184425 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:15.951379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:15.951344 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:16.749020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:16.748982 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:16.868725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:16.868599 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:16.966611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:16.966536 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:16.991763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:16.991729 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:28:17.401131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:17.401087 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:18.947299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:18.947243 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:19.129584 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.129549 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:19.290687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.290595 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr"] Oct 30 18:28:19.310498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.309269 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" containerID="cri-o://7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a" gracePeriod=30 Oct 30 18:28:19.314607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.314484 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:19.314893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.314874 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft"] Oct 30 18:28:19.357535 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.352992 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf"] Oct 30 18:28:19.357535 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.353234 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" containerID="cri-o://4be7100ade67cf05006cba0b406a6647d3f27abebb57bb627d449e327748187c" gracePeriod=30 Oct 30 18:28:19.383662 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.378627 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" containerID="cri-o://45da120f5ee93f4c9ad309b6ce3d7bebce1bb59f4f28b5b3861b836f8d9a1653" gracePeriod=30 Oct 30 18:28:19.420654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.392984 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd"] Oct 30 18:28:19.420654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.393233 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" containerID="cri-o://0e8103da125a531de6c8ae8ef523c201a48abfd1b72f86d9d78a3417adc2c474" gracePeriod=30 Oct 30 18:28:19.471751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.470074 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:19.699563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.698914 5624 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:28:19.699563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:19.698984 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:28:20.190423 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:20.190182 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:28:21.005959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:20.997016 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:21.468959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:21.468015 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:21.573961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:21.564970 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:21.657272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:21.657230 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:22.054759 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.054721 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" containerID="cri-o://306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030" gracePeriod=45 Oct 30 18:28:22.436246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.436214 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:28:22.521160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.521126 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-kube-api-access-fcwhq\" (UniqueName: \"kubernetes.io/projected/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-kube-api-access-fcwhq\") pod \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " Oct 30 18:28:22.521635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.521353 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-model-cache\") pod \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " Oct 30 18:28:22.521635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.521415 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-home\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-home\") pod \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " Oct 30 18:28:22.521635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.521464 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tmp-dir\") pod \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " Oct 30 18:28:22.521635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.521526 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-dshm\") pod \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " Oct 30 18:28:22.521864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.521644 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tls-certs\") pod \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\" (UID: \"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8\") " Oct 30 18:28:22.521864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.521842 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-home" pod "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" (UID: "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.522154 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.522126 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tmp-dir" pod "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" (UID: "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.522539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.522493 5624 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-home\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.522539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.522511 5624 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tmp-dir\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.522752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.522724 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-model-cache" pod "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" (UID: "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.524883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.524853 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tls-certs" pod "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" (UID: "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:22.525326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.525303 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-kube-api-access-fcwhq" pod "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" (UID: "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8"). InnerVolumeSpecName "kube-api-access-fcwhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:22.527217 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.527189 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-dshm" pod "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" (UID: "16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.624635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.624601 5624 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-dshm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.624635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.624625 5624 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-tls-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.624635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.624636 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcwhq\" (UniqueName: \"kubernetes.io/projected/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-kube-api-access-fcwhq\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.624635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.624646 5624 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8-model-cache\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.886694 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.886650 5624 generic.go:356] "Generic (PLEG): container finished" podID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerID="306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030" exitCode=0 Oct 30 18:28:22.886883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.886703 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" Oct 30 18:28:22.886883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.886725 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" event={"ID":"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8","Type":"ContainerDied","Data":"306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030"} Oct 30 18:28:22.886883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.886782 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2" event={"ID":"16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8","Type":"ContainerDied","Data":"c48e993f89c3ddf3f32da5d19c3032e653b2dd47e72e39339c37da4132490d7e"} Oct 30 18:28:22.886883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.886803 5624 scope.go:122] "RemoveContainer" containerID="306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030" Oct 30 18:28:22.901792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.901772 5624 scope.go:122] "RemoveContainer" containerID="306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030" Oct 30 18:28:22.902191 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:22.902157 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030\": container with ID starting with 306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030 not found: ID does not exist" containerID="306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030" Oct 30 18:28:22.902323 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.902201 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030"} err="failed to get container status \"306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030\": rpc error: code = NotFound desc = could not find container \"306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030\": container with ID starting with 306967cd95684af32b03b9dedf9d1e54f479d30bc12737a360abaae708815030 not found: ID does not exist" Oct 30 18:28:22.944727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.944708 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2"] Oct 30 18:28:22.949648 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:22.949628 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/simulator-w6-ef13cc-kserve-5b57d8bff8-lcfw2"] Oct 30 18:28:23.240425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:23.240352 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:23.795866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:23.795820 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" path="/var/lib/kubelet/pods/16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8/volumes" Oct 30 18:28:24.112664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.112583 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:24.212696 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.212655 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:24.454991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.451413 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p"] Oct 30 18:28:24.454991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.451738 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" containerID="cri-o://4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711" gracePeriod=30 Oct 30 18:28:24.482369 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.482339 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:24.847648 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.839000 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:28:24.921578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.921545 5624 generic.go:356] "Generic (PLEG): container finished" podID="7f87e377-56d3-4931-8d5e-e9d479166009" containerID="0e8103da125a531de6c8ae8ef523c201a48abfd1b72f86d9d78a3417adc2c474" exitCode=0 Oct 30 18:28:24.921731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.921675 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" event={"ID":"7f87e377-56d3-4931-8d5e-e9d479166009","Type":"ContainerDied","Data":"0e8103da125a531de6c8ae8ef523c201a48abfd1b72f86d9d78a3417adc2c474"} Oct 30 18:28:24.925199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.924180 5624 generic.go:356] "Generic (PLEG): container finished" podID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerID="45da120f5ee93f4c9ad309b6ce3d7bebce1bb59f4f28b5b3861b836f8d9a1653" exitCode=0 Oct 30 18:28:24.925199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.924239 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" event={"ID":"47e369dc-6ca5-45f2-9c24-00d8a7421900","Type":"ContainerDied","Data":"45da120f5ee93f4c9ad309b6ce3d7bebce1bb59f4f28b5b3861b836f8d9a1653"} Oct 30 18:28:24.925199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.924272 5624 scope.go:122] "RemoveContainer" containerID="3e444475098266a85924106c0ef783198bb61b7068a28747ef16e3c1ca63d06e" Oct 30 18:28:24.926704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.926676 5624 generic.go:356] "Generic (PLEG): container finished" podID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerID="7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a" exitCode=0 Oct 30 18:28:24.926924 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.926901 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" event={"ID":"f5b4e3bb-6754-4431-bd4a-c2b1512457ba","Type":"ContainerDied","Data":"7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a"} Oct 30 18:28:24.927022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.926976 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" event={"ID":"f5b4e3bb-6754-4431-bd4a-c2b1512457ba","Type":"ContainerDied","Data":"934f1bd7da4ee3ab815c9f0e4428052b68527382488969f2c03da913ca193d02"} Oct 30 18:28:24.927097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.927074 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr" Oct 30 18:28:24.927750 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.927702 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:24.929468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.929393 5624 generic.go:356] "Generic (PLEG): container finished" podID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerID="4be7100ade67cf05006cba0b406a6647d3f27abebb57bb627d449e327748187c" exitCode=0 Oct 30 18:28:24.929555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.929466 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" event={"ID":"97dcb1f4-4b8e-44c0-a306-a94cafcca635","Type":"ContainerDied","Data":"4be7100ade67cf05006cba0b406a6647d3f27abebb57bb627d449e327748187c"} Oct 30 18:28:24.944409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.944386 5624 scope.go:122] "RemoveContainer" containerID="7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a" Oct 30 18:28:24.959351 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.959337 5624 scope.go:122] "RemoveContainer" containerID="a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6" Oct 30 18:28:24.972079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.972060 5624 scope.go:122] "RemoveContainer" containerID="7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a" Oct 30 18:28:24.972415 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:24.972375 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a\": container with ID starting with 7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a not found: ID does not exist" containerID="7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a" Oct 30 18:28:24.972524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.972427 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a"} err="failed to get container status \"7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a\": rpc error: code = NotFound desc = could not find container \"7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a\": container with ID starting with 7a41dcade8601a5d6552108d753b371d56bd48212412a951020534ce866abd7a not found: ID does not exist" Oct 30 18:28:24.972524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.972448 5624 scope.go:122] "RemoveContainer" containerID="a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6" Oct 30 18:28:24.972822 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:24.972739 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6\": container with ID starting with a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6 not found: ID does not exist" containerID="a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6" Oct 30 18:28:24.972822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.972766 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6"} err="failed to get container status \"a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6\": rpc error: code = NotFound desc = could not find container \"a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6\": container with ID starting with a154dc75217286cf01758faa487814fb218aa88118fca04cb9720c33d12ef9c6 not found: ID does not exist" Oct 30 18:28:24.972822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.972785 5624 scope.go:122] "RemoveContainer" containerID="4be7100ade67cf05006cba0b406a6647d3f27abebb57bb627d449e327748187c" Oct 30 18:28:24.985498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.985471 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-certs\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.985624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.985557 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-podinfo\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.985624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.985595 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-socket\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.985746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.985625 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-data\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.985746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.985705 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-token\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.985851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.985763 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-credential-socket\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.985918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.985895 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-envoy\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.987123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.986063 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istiod-ca-cert\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.987123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.986191 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-kube-api-access-krjfj\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-kube-api-access-krjfj\") pod \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\" (UID: \"f5b4e3bb-6754-4431-bd4a-c2b1512457ba\") " Oct 30 18:28:24.987123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.986435 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-data" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.987123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.986905 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.987123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.987056 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-certs" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.987677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.987659 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-socket" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.987798 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.987737 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istiod-ca-cert" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:24.988001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.987965 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-credential-socket" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.988001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.989436 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:24.988001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:24.989760 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-kube-api-access-krjfj" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "kube-api-access-krjfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.001193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.001158 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-token" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.005165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.005137 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-envoy" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.005273 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.005209 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-podinfo" pod "f5b4e3bb-6754-4431-bd4a-c2b1512457ba" (UID: "f5b4e3bb-6754-4431-bd4a-c2b1512457ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.079813 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087393 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-certs\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087461 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-token\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-token\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087528 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-kube-api-access-ccmwp\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-kube-api-access-ccmwp\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087567 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-data\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087603 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-socket\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087656 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f87e377-56d3-4931-8d5e-e9d479166009-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f87e377-56d3-4931-8d5e-e9d479166009-istio-podinfo\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087717 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-podinfo\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087821 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f87e377-56d3-4931-8d5e-e9d479166009-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f87e377-56d3-4931-8d5e-e9d479166009-istiod-ca-cert\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087864 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-credential-socket\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.087910 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-envoy\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088034 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-credential-socket\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088069 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istiod-ca-cert\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088116 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-data\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088174 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-istio-token\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-istio-token\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088224 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-kube-api-access-ggxdj\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-kube-api-access-ggxdj\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088297 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-socket\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088361 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-envoy\") pod \"7f87e377-56d3-4931-8d5e-e9d479166009\" (UID: \"7f87e377-56d3-4931-8d5e-e9d479166009\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088432 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-certs\") pod \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\" (UID: \"97dcb1f4-4b8e-44c0-a306-a94cafcca635\") " Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.088753 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istiod-ca-cert" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089372 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089394 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krjfj\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-kube-api-access-krjfj\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089410 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089424 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089436 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089448 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089461 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089474 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.089615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089487 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b4e3bb-6754-4431-bd4a-c2b1512457ba-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.091277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.089963 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-certs" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.091277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.090177 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-data" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.092823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.091891 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-socket" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.092823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.092032 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-envoy" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.092823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.092237 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-credential-socket" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.092823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.092287 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-credential-socket" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.092823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.092575 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f87e377-56d3-4931-8d5e-e9d479166009-istiod-ca-cert" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:25.093695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.093545 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-data" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.093979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.093814 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-socket" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.094361 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.094093 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-certs" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.096852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.096598 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-kube-api-access-ccmwp" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "kube-api-access-ccmwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.096852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.096694 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-istio-token" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.096852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.096771 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-envoy" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.096852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.096826 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f87e377-56d3-4931-8d5e-e9d479166009-istio-podinfo" pod "7f87e377-56d3-4931-8d5e-e9d479166009" (UID: "7f87e377-56d3-4931-8d5e-e9d479166009"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.097567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.097535 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-podinfo" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.098367 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.098031 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-token" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.098755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.098728 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-kube-api-access-ggxdj" pod "97dcb1f4-4b8e-44c0-a306-a94cafcca635" (UID: "97dcb1f4-4b8e-44c0-a306-a94cafcca635"). InnerVolumeSpecName "kube-api-access-ggxdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.193738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193671 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.193738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193724 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.193738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193738 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193751 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193764 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193775 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxdj\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-kube-api-access-ggxdj\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193786 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193800 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193813 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97dcb1f4-4b8e-44c0-a306-a94cafcca635-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193825 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193837 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193849 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccmwp\" (UniqueName: \"kubernetes.io/projected/7f87e377-56d3-4931-8d5e-e9d479166009-kube-api-access-ccmwp\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193861 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193875 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f87e377-56d3-4931-8d5e-e9d479166009-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193886 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f87e377-56d3-4931-8d5e-e9d479166009-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193901 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97dcb1f4-4b8e-44c0-a306-a94cafcca635-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.194169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.193914 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f87e377-56d3-4931-8d5e-e9d479166009-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.196826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.196800 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:28:25.286801 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.286765 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr"] Oct 30 18:28:25.289683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.289648 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:25.295205 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295178 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-credential-socket\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.295318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295237 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e369dc-6ca5-45f2-9c24-00d8a7421900-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e369dc-6ca5-45f2-9c24-00d8a7421900-istiod-ca-cert\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.295431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295334 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-kube-api-access-m7rbf\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-kube-api-access-m7rbf\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.295431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295391 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-socket\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.295541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295455 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-data\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.295541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295522 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-token\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-token\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.295643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295575 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-podinfo\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.295700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295643 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-certs\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.295753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.295705 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-envoy\") pod \"47e369dc-6ca5-45f2-9c24-00d8a7421900\" (UID: \"47e369dc-6ca5-45f2-9c24-00d8a7421900\") " Oct 30 18:28:25.296432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.296406 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-q5dbr"] Oct 30 18:28:25.296656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.296635 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-certs" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.296734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.296582 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-socket" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.296870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.296849 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-data" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.297298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.296955 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e369dc-6ca5-45f2-9c24-00d8a7421900-istiod-ca-cert" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:25.297298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.297193 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-credential-socket" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.300266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.300208 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-podinfo" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.300266 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.300239 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-kube-api-access-m7rbf" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "kube-api-access-m7rbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.300564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.300544 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-token" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.301455 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.301436 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-envoy" pod "47e369dc-6ca5-45f2-9c24-00d8a7421900" (UID: "47e369dc-6ca5-45f2-9c24-00d8a7421900"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.397549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397518 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.397549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397550 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.397747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397562 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.397747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397573 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.397747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397585 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.397747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397597 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e369dc-6ca5-45f2-9c24-00d8a7421900-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.397747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397611 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7rbf\" (UniqueName: \"kubernetes.io/projected/47e369dc-6ca5-45f2-9c24-00d8a7421900-kube-api-access-m7rbf\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.397747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397624 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.397747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.397636 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e369dc-6ca5-45f2-9c24-00d8a7421900-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.793281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.793250 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" path="/var/lib/kubelet/pods/f5b4e3bb-6754-4431-bd4a-c2b1512457ba/volumes" Oct 30 18:28:25.862287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.862231 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:25.939578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.939551 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" Oct 30 18:28:25.939760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.939550 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft" event={"ID":"47e369dc-6ca5-45f2-9c24-00d8a7421900","Type":"ContainerDied","Data":"85638057edf6383124e3ef0183fd01a7001032c3e28f42b318e7f90a14996697"} Oct 30 18:28:25.939760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.939692 5624 scope.go:122] "RemoveContainer" containerID="45da120f5ee93f4c9ad309b6ce3d7bebce1bb59f4f28b5b3861b836f8d9a1653" Oct 30 18:28:25.942197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.942171 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" event={"ID":"97dcb1f4-4b8e-44c0-a306-a94cafcca635","Type":"ContainerDied","Data":"fe1f6cbba58b6633da57a2b60feff6043e6ab5c6671f1b32ae135b26c2d5acc5"} Oct 30 18:28:25.942288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.942268 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf" Oct 30 18:28:25.945398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.945373 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" event={"ID":"7f87e377-56d3-4931-8d5e-e9d479166009","Type":"ContainerDied","Data":"0360978c4132d15a8c33c1b737614213cb85f39673d7d65de70b7802a5060db2"} Oct 30 18:28:25.945560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:25.945400 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd" Oct 30 18:28:26.009726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:26.008844 5624 scope.go:122] "RemoveContainer" containerID="0e8103da125a531de6c8ae8ef523c201a48abfd1b72f86d9d78a3417adc2c474" Oct 30 18:28:26.070639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:26.070584 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:26.104496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:26.104463 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft"] Oct 30 18:28:26.131306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:26.131177 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-jz7ft"] Oct 30 18:28:26.172808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:26.172631 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf"] Oct 30 18:28:26.177631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:26.177605 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-49vcf"] Oct 30 18:28:26.206224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:26.206193 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd"] Oct 30 18:28:26.210884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:26.210862 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-4crfd"] Oct 30 18:28:27.012424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:27.012379 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:27.112832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:27.112800 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:27.212319 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:27.212285 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:27.793168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:27.793138 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" path="/var/lib/kubelet/pods/47e369dc-6ca5-45f2-9c24-00d8a7421900/volumes" Oct 30 18:28:27.793707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:27.793692 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" path="/var/lib/kubelet/pods/7f87e377-56d3-4931-8d5e-e9d479166009/volumes" Oct 30 18:28:27.794212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:27.794194 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" path="/var/lib/kubelet/pods/97dcb1f4-4b8e-44c0-a306-a94cafcca635/volumes" Oct 30 18:28:28.787492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:28.787453 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:29.131790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.131725 5624 patch_prober.go:29] interesting pod/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:28:29.132238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.132089 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:28:29.745272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.745220 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:28:29.782589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.782554 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-podinfo\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.782746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.782631 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-data\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.782746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.782694 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-credential-socket\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.782879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.782811 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-token\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-token\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.783104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.782998 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-credential-socket" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:29.783104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.783073 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-data" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:29.783321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.783234 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-certs\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.783321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.783269 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-socket\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.783321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.783315 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istiod-ca-cert\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.783541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.783353 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-envoy\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.783657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.783637 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-kube-api-access-gck2m\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-kube-api-access-gck2m\") pod \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\" (UID: \"9e3d9000-f1f2-450b-9da7-c1de20b34b77\") " Oct 30 18:28:29.783657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.783645 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-socket" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:29.784183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.783920 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istiod-ca-cert" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:29.784183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.784018 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-certs" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:29.784353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.784324 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.784353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.784344 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.785539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.784358 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.785539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.784372 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.785539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.784384 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.785539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.785286 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-podinfo" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:29.785918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.785890 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-token" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:29.786359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.786334 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-envoy" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:29.786699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.786669 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-kube-api-access-gck2m" pod "9e3d9000-f1f2-450b-9da7-c1de20b34b77" (UID: "9e3d9000-f1f2-450b-9da7-c1de20b34b77"). InnerVolumeSpecName "kube-api-access-gck2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:29.885423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.885392 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.885423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.885418 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gck2m\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-kube-api-access-gck2m\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.885423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.885427 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.885832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.885436 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3d9000-f1f2-450b-9da7-c1de20b34b77-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:29.919200 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.919178 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:29.988786 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.988755 5624 generic.go:356] "Generic (PLEG): container finished" podID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerID="4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711" exitCode=0 Oct 30 18:28:29.988984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.988836 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" event={"ID":"9e3d9000-f1f2-450b-9da7-c1de20b34b77","Type":"ContainerDied","Data":"4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711"} Oct 30 18:28:29.988984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.988862 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" Oct 30 18:28:29.988984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.988890 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p" event={"ID":"9e3d9000-f1f2-450b-9da7-c1de20b34b77","Type":"ContainerDied","Data":"944806e00af476008849a8f2c0856748868f207a2dce0c6e353815afd51bde57"} Oct 30 18:28:29.988984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:29.988913 5624 scope.go:122] "RemoveContainer" containerID="4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711" Oct 30 18:28:30.001336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.001322 5624 scope.go:122] "RemoveContainer" containerID="0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f" Oct 30 18:28:30.012123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.012107 5624 scope.go:122] "RemoveContainer" containerID="4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711" Oct 30 18:28:30.012404 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:30.012379 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711\": container with ID starting with 4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711 not found: ID does not exist" containerID="4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711" Oct 30 18:28:30.012494 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.012410 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711"} err="failed to get container status \"4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711\": rpc error: code = NotFound desc = could not find container \"4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711\": container with ID starting with 4b1b4d949a6b518d0d16c57abb86cd0c91ef6b433c98c5e66c573370f17d2711 not found: ID does not exist" Oct 30 18:28:30.012494 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.012427 5624 scope.go:122] "RemoveContainer" containerID="0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f" Oct 30 18:28:30.012747 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:30.012726 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f\": container with ID starting with 0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f not found: ID does not exist" containerID="0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f" Oct 30 18:28:30.012797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.012753 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f"} err="failed to get container status \"0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f\": rpc error: code = NotFound desc = could not find container \"0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f\": container with ID starting with 0500ba7d7f43ff1d66ba5da7e3f500a9d3fcca398974a70d9c626c6c4599925f not found: ID does not exist" Oct 30 18:28:30.037922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.037892 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:30.053116 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.053088 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p"] Oct 30 18:28:30.060165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.060141 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-227c01-openshift-default-7987fbf485-8lg2p"] Oct 30 18:28:30.149687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.149654 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:30.459336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.459310 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:30.722579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.722499 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:30.821761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:30.821729 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:31.814956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:31.814693 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" path="/var/lib/kubelet/pods/9e3d9000-f1f2-450b-9da7-c1de20b34b77/volumes" Oct 30 18:28:31.922965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:31.920980 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:32.020431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.020389 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:32.116639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.116552 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r"] Oct 30 18:28:32.117561 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117544 5624 cpu_manager.go:414] "containerMap: removing container" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" Oct 30 18:28:32.117611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117564 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" Oct 30 18:28:32.117611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117583 5624 cpu_manager.go:414] "containerMap: removing container" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" Oct 30 18:28:32.117611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117591 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" Oct 30 18:28:32.117611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117603 5624 cpu_manager.go:414] "containerMap: removing container" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117612 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117626 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117633 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117649 5624 cpu_manager.go:414] "containerMap: removing container" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117656 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117670 5624 cpu_manager.go:414] "containerMap: removing container" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117677 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117702 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" Oct 30 18:28:32.117748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117708 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117892 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117905 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117917 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="97dcb1f4-4b8e-44c0-a306-a94cafcca635" containerName="istio-proxy" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117926 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f87e377-56d3-4931-8d5e-e9d479166009" containerName="istio-proxy" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117948 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117956 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117967 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117975 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d4b0da-7d2b-49c3-b2c3-9c450f7a79a8" containerName="main" Oct 30 18:28:32.118044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.117986 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3d9000-f1f2-450b-9da7-c1de20b34b77" containerName="istio-proxy" Oct 30 18:28:32.123696 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.123673 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:32.141957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.139316 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1b0848\"" Oct 30 18:28:32.141957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.139430 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1b0848-dockercfg-gllp7\"" Oct 30 18:28:32.141957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.140631 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r"] Oct 30 18:28:32.252957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.247907 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87290e0a-f02f-4aaf-84e8-3526e1b63c54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r\" (UID: \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:32.252957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.248103 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qksp\" (UniqueName: \"kubernetes.io/projected/87290e0a-f02f-4aaf-84e8-3526e1b63c54-kube-api-access-5qksp\") pod \"payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r\" (UID: \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:32.362008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.357715 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87290e0a-f02f-4aaf-84e8-3526e1b63c54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r\" (UID: \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:32.393085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.387163 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qksp\" (UniqueName: \"kubernetes.io/projected/87290e0a-f02f-4aaf-84e8-3526e1b63c54-kube-api-access-5qksp\") pod \"payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r\" (UID: \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:32.417427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.417378 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87290e0a-f02f-4aaf-84e8-3526e1b63c54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r\" (UID: \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:32.447779 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.447648 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qksp\" (UniqueName: \"kubernetes.io/projected/87290e0a-f02f-4aaf-84e8-3526e1b63c54-kube-api-access-5qksp\") pod \"payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r\" (UID: \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:32.514191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.513997 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:32.741965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:32.741872 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:33.077766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:33.077348 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r"] Oct 30 18:28:33.902957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:33.902416 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:34.044547 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:34.044517 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" event={"ID":"87290e0a-f02f-4aaf-84e8-3526e1b63c54","Type":"ContainerStarted","Data":"57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181"} Oct 30 18:28:34.044772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:34.044759 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:34.044867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:34.044855 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" event={"ID":"87290e0a-f02f-4aaf-84e8-3526e1b63c54","Type":"ContainerStarted","Data":"6cdae25d33f238c9ed57f2585b0d51113be32c489e8f50e9283fc84aeae742a1"} Oct 30 18:28:34.097581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:34.097520 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" podStartSLOduration=2.097500694 podStartE2EDuration="2.097500694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:34.093441034 +0000 UTC m=+1608.481969148" watchObservedRunningTime="2026-10-30 18:28:34.097500694 +0000 UTC m=+1608.486028811" Oct 30 18:28:34.461448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:34.460801 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:34.561484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:34.561452 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:34.709750 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:34.709717 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:35.159467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:35.159430 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:36.437446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:36.437413 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:37.901842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.901809 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt"] Oct 30 18:28:37.904062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.904019 5624 cpu_manager.go:414] "containerMap: removing container" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" Oct 30 18:28:37.904292 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.904275 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b4e3bb-6754-4431-bd4a-c2b1512457ba" containerName="istio-proxy" Oct 30 18:28:37.904462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.904449 5624 cpu_manager.go:414] "containerMap: removing container" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" Oct 30 18:28:37.904589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.904562 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e369dc-6ca5-45f2-9c24-00d8a7421900" containerName="istio-proxy" Oct 30 18:28:37.913169 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.913146 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:37.929241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.929218 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt"] Oct 30 18:28:37.951542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.951500 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:37.951686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.951678 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:28:37.963211 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.963182 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqkj\" (UniqueName: \"kubernetes.io/projected/9c29e32f-861c-48c3-946b-5ffa93867689-kube-api-access-nrqkj\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:37.963465 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.963446 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:37.963662 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:37.963649 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:38.064593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.064555 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:38.064789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.064695 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqkj\" (UniqueName: \"kubernetes.io/projected/9c29e32f-861c-48c3-946b-5ffa93867689-kube-api-access-nrqkj\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:38.064789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.064767 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:38.065319 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.065292 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:38.067414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.067382 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:38.088270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.088239 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqkj\" (UniqueName: \"kubernetes.io/projected/9c29e32f-861c-48c3-946b-5ffa93867689-kube-api-access-nrqkj\") pod \"maas-api-revoke-sub-0206e564-vb9pt\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:38.240719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.240639 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:38.278346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.278309 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:38.564809 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:38.564779 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt"] Oct 30 18:28:39.093137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:39.093103 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" event={"ID":"9c29e32f-861c-48c3-946b-5ffa93867689","Type":"ContainerStarted","Data":"e59884f7f31900fd47e991bb1b2b9b046a15cc63c30ca0db55e3a1768743153c"} Oct 30 18:28:39.093549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:39.093164 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" event={"ID":"9c29e32f-861c-48c3-946b-5ffa93867689","Type":"ContainerStarted","Data":"6ad7bb016732f2682c9c692d0898aa4d6adf1fb867cac38d0f3f9dab0d389dd6"} Oct 30 18:28:39.134471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:39.134415 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" podStartSLOduration=2.134397492 podStartE2EDuration="2.134397492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:39.134084151 +0000 UTC m=+1613.522612270" watchObservedRunningTime="2026-10-30 18:28:39.134397492 +0000 UTC m=+1613.522925610" Oct 30 18:28:39.565106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:39.565067 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:39.665630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:39.665591 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:40.036542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:40.036506 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:40.123788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:40.123754 5624 generic.go:356] "Generic (PLEG): container finished" podID="9c29e32f-861c-48c3-946b-5ffa93867689" containerID="e59884f7f31900fd47e991bb1b2b9b046a15cc63c30ca0db55e3a1768743153c" exitCode=0 Oct 30 18:28:40.124401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:40.123883 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" event={"ID":"9c29e32f-861c-48c3-946b-5ffa93867689","Type":"ContainerDied","Data":"e59884f7f31900fd47e991bb1b2b9b046a15cc63c30ca0db55e3a1768743153c"} Oct 30 18:28:41.298856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.298827 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:41.428778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.428685 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-openshift-ca\") pod \"9c29e32f-861c-48c3-946b-5ffa93867689\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " Oct 30 18:28:41.428778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.428735 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c29e32f-861c-48c3-946b-5ffa93867689-kube-api-access-nrqkj\" (UniqueName: \"kubernetes.io/projected/9c29e32f-861c-48c3-946b-5ffa93867689-kube-api-access-nrqkj\") pod \"9c29e32f-861c-48c3-946b-5ffa93867689\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " Oct 30 18:28:41.429076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.428854 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-xks-ca\") pod \"9c29e32f-861c-48c3-946b-5ffa93867689\" (UID: \"9c29e32f-861c-48c3-946b-5ffa93867689\") " Oct 30 18:28:41.429175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.429147 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-openshift-ca" pod "9c29e32f-861c-48c3-946b-5ffa93867689" (UID: "9c29e32f-861c-48c3-946b-5ffa93867689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:41.429598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.429572 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.431619 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.431589 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-xks-ca" pod "9c29e32f-861c-48c3-946b-5ffa93867689" (UID: "9c29e32f-861c-48c3-946b-5ffa93867689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:41.431723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.431596 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c29e32f-861c-48c3-946b-5ffa93867689-kube-api-access-nrqkj" pod "9c29e32f-861c-48c3-946b-5ffa93867689" (UID: "9c29e32f-861c-48c3-946b-5ffa93867689"). InnerVolumeSpecName "kube-api-access-nrqkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:41.530965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.530909 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqkj\" (UniqueName: \"kubernetes.io/projected/9c29e32f-861c-48c3-946b-5ffa93867689-kube-api-access-nrqkj\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.530965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.530949 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c29e32f-861c-48c3-946b-5ffa93867689-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.863210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.863179 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:41.985283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:41.985249 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.089542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.089440 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.150088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.150050 5624 generic.go:356] "Generic (PLEG): container finished" podID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerID="a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9" exitCode=0 Oct 30 18:28:42.150313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.150289 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" event={"ID":"2525bc9c-24dd-4ee0-afe9-2d79c0266d41","Type":"ContainerDied","Data":"a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9"} Oct 30 18:28:42.150853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.150815 5624 scope.go:122] "RemoveContainer" containerID="a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9" Oct 30 18:28:42.152545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.152522 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" Oct 30 18:28:42.152651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.152559 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt" event={"ID":"9c29e32f-861c-48c3-946b-5ffa93867689","Type":"ContainerDied","Data":"6ad7bb016732f2682c9c692d0898aa4d6adf1fb867cac38d0f3f9dab0d389dd6"} Oct 30 18:28:42.152651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.152596 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ad7bb016732f2682c9c692d0898aa4d6adf1fb867cac38d0f3f9dab0d389dd6" Oct 30 18:28:42.152796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.152772 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:42.152851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.152839 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:42.152893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.152879 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:28:42.273631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.273601 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.378306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.378279 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7"] Oct 30 18:28:42.395928 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.395610 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt"] Oct 30 18:28:42.396253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.396220 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" containerID="cri-o://9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e" gracePeriod=30 Oct 30 18:28:42.411163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.411097 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.636680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.636597 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:42.928210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:42.928121 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:43.162331 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:43.162289 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" event={"ID":"2525bc9c-24dd-4ee0-afe9-2d79c0266d41","Type":"ContainerStarted","Data":"4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4"} Oct 30 18:28:43.162508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:43.162344 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerName="istio-proxy" containerID="cri-o://4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4" gracePeriod=30 Oct 30 18:28:43.602231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:43.602198 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:44.001870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:44.001835 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:44.148074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:44.146258 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:44.312044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:44.311920 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:44.573673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:44.573592 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:45.057276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:45.057247 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:28:45.626900 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:45.626867 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:45.866756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:45.866721 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:47.473556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.473526 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:47.733629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.733598 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:28:47.870794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.870762 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-certs\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871074 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-envoy\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871282 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-credential-socket\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871334 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8626ce00-6a12-48e5-a761-c20623fd714a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626ce00-6a12-48e5-a761-c20623fd714a-istiod-ca-cert\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871466 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871452 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-istio-token\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-istio-token\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871503 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-kube-api-access-tfllm\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-kube-api-access-tfllm\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871573 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8626ce00-6a12-48e5-a761-c20623fd714a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626ce00-6a12-48e5-a761-c20623fd714a-istio-podinfo\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871644 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871586 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-certs" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.871701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871663 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-data\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871773 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-socket\") pod \"8626ce00-6a12-48e5-a761-c20623fd714a\" (UID: \"8626ce00-6a12-48e5-a761-c20623fd714a\") " Oct 30 18:28:47.871883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.871866 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8626ce00-6a12-48e5-a761-c20623fd714a-istiod-ca-cert" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:47.872291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.872271 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.872392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.872295 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8626ce00-6a12-48e5-a761-c20623fd714a-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.872542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.872522 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-socket" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.873051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.873020 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-credential-socket" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.873719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.873696 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-data" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.876763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.876741 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-istio-token" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.876899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.876784 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8626ce00-6a12-48e5-a761-c20623fd714a-istio-podinfo" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:47.877062 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.877034 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-kube-api-access-tfllm" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "kube-api-access-tfllm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.881229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.881208 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-envoy" pod "8626ce00-6a12-48e5-a761-c20623fd714a" (UID: "8626ce00-6a12-48e5-a761-c20623fd714a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.905604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.905579 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2"] Oct 30 18:28:47.927640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.927608 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm"] Oct 30 18:28:47.973800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.973721 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.973800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.973760 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.973800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.973776 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfllm\" (UniqueName: \"kubernetes.io/projected/8626ce00-6a12-48e5-a761-c20623fd714a-kube-api-access-tfllm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.973800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.973789 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8626ce00-6a12-48e5-a761-c20623fd714a-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.973800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.973805 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.974231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.973819 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.974231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.973832 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8626ce00-6a12-48e5-a761-c20623fd714a-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.999059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:47.999028 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:48.238712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.238629 5624 generic.go:356] "Generic (PLEG): container finished" podID="8626ce00-6a12-48e5-a761-c20623fd714a" containerID="9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e" exitCode=0 Oct 30 18:28:48.238712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.238686 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" Oct 30 18:28:48.238884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.238715 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" event={"ID":"8626ce00-6a12-48e5-a761-c20623fd714a","Type":"ContainerDied","Data":"9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e"} Oct 30 18:28:48.238884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.238770 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt" event={"ID":"8626ce00-6a12-48e5-a761-c20623fd714a","Type":"ContainerDied","Data":"cb48ef7807c0a29a735af15b17f4f94028e8b0b909a4aacc171ea96d968bc3c8"} Oct 30 18:28:48.238884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.238793 5624 scope.go:122] "RemoveContainer" containerID="9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e" Oct 30 18:28:48.276882 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.276864 5624 scope.go:122] "RemoveContainer" containerID="6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25" Oct 30 18:28:48.288246 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.288222 5624 scope.go:122] "RemoveContainer" containerID="9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e" Oct 30 18:28:48.288524 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:48.288501 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e\": container with ID starting with 9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e not found: ID does not exist" containerID="9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e" Oct 30 18:28:48.288617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.288535 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e"} err="failed to get container status \"9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e\": rpc error: code = NotFound desc = could not find container \"9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e\": container with ID starting with 9003348b1e3114afa9609de0abe51d347cc8a2b9f0db60719f132e144125e48e not found: ID does not exist" Oct 30 18:28:48.288617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.288559 5624 scope.go:122] "RemoveContainer" containerID="6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25" Oct 30 18:28:48.288882 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:48.288862 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25\": container with ID starting with 6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25 not found: ID does not exist" containerID="6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25" Oct 30 18:28:48.288994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.288890 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25"} err="failed to get container status \"6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25\": rpc error: code = NotFound desc = could not find container \"6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25\": container with ID starting with 6dde8650e0df421cc66b42b2dfe796f86ab088528952453aa2db22e07b293b25 not found: ID does not exist" Oct 30 18:28:48.299432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.299398 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt"] Oct 30 18:28:48.311060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.311032 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-8mzlt"] Oct 30 18:28:48.329546 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.329515 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4"] Oct 30 18:28:48.367677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.367645 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5"] Oct 30 18:28:48.461148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.461109 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:48.484135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.484104 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:28:48.553971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.553878 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:48.693456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.693429 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-kube-api-access-8sd22\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-kube-api-access-8sd22\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.693732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.693490 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-podinfo\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.693732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.693642 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-credential-socket\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.693732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.693689 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istiod-ca-cert\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.693997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.693779 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-data\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.693997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.693847 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-envoy\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.693997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.693922 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-token\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-token\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.693997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.693992 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-socket\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.694263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694064 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-certs\") pod \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\" (UID: \"2525bc9c-24dd-4ee0-afe9-2d79c0266d41\") " Oct 30 18:28:48.694263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694107 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-credential-socket" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.694263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694186 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-data" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.694510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694474 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istiod-ca-cert" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:48.694681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694660 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-socket" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.694788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694767 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.694857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694794 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.694857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694807 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-certs" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.694857 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.694808 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.697998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.697376 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-kube-api-access-8sd22" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "kube-api-access-8sd22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:48.697998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.697581 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-envoy" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:48.697998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.697793 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-podinfo" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:48.697998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.697849 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-token" pod "2525bc9c-24dd-4ee0-afe9-2d79c0266d41" (UID: "2525bc9c-24dd-4ee0-afe9-2d79c0266d41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:48.796113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.796075 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sd22\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-kube-api-access-8sd22\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.796113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.796112 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.796296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.796126 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.796296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.796138 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.796296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.796149 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.796296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:48.796160 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2525bc9c-24dd-4ee0-afe9-2d79c0266d41-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:49.096841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.096803 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:49.261960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.259902 5624 generic.go:356] "Generic (PLEG): container finished" podID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerID="4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4" exitCode=0 Oct 30 18:28:49.261960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.260128 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" event={"ID":"2525bc9c-24dd-4ee0-afe9-2d79c0266d41","Type":"ContainerDied","Data":"4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4"} Oct 30 18:28:49.261960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.260178 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" event={"ID":"2525bc9c-24dd-4ee0-afe9-2d79c0266d41","Type":"ContainerDied","Data":"8f4b507247e93eda430067214e6168866628bd2d3fcf79ee5e3eaf39c91f6bf9"} Oct 30 18:28:49.261960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.260206 5624 scope.go:122] "RemoveContainer" containerID="4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4" Oct 30 18:28:49.261960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.260318 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7" Oct 30 18:28:49.351181 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.350518 5624 scope.go:122] "RemoveContainer" containerID="a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9" Oct 30 18:28:49.362096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.353329 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7"] Oct 30 18:28:49.369492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.369470 5624 scope.go:122] "RemoveContainer" containerID="4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4" Oct 30 18:28:49.370397 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:49.370372 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4\": container with ID starting with 4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4 not found: ID does not exist" containerID="4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4" Oct 30 18:28:49.370510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.370403 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4"} err="failed to get container status \"4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4\": rpc error: code = NotFound desc = could not find container \"4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4\": container with ID starting with 4024482fe77beca0c2e23f5506ce1caa89d5ccb40a97de8175619a92cc9a10f4 not found: ID does not exist" Oct 30 18:28:49.370510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.370420 5624 scope.go:122] "RemoveContainer" containerID="a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9" Oct 30 18:28:49.370666 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:49.370646 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9\": container with ID starting with a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9 not found: ID does not exist" containerID="a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9" Oct 30 18:28:49.370734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.370671 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9"} err="failed to get container status \"a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9\": rpc error: code = NotFound desc = could not find container \"a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9\": container with ID starting with a7a1cf6cab16803eac2b073a6a65875104fa8a362df0ed169eb331917e4a18e9 not found: ID does not exist" Oct 30 18:28:49.372784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.372761 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-km5x7"] Oct 30 18:28:49.795555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.795524 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" path="/var/lib/kubelet/pods/2525bc9c-24dd-4ee0-afe9-2d79c0266d41/volumes" Oct 30 18:28:49.796159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:49.796144 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" path="/var/lib/kubelet/pods/8626ce00-6a12-48e5-a761-c20623fd714a/volumes" Oct 30 18:28:49.919112 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:28:49.919072 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:28:51.166020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.158520 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24"] Oct 30 18:28:51.166515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.166154 5624 cpu_manager.go:414] "containerMap: removing container" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerName="istio-proxy" Oct 30 18:28:51.166515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.166179 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerName="istio-proxy" Oct 30 18:28:51.166515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.166209 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9c29e32f-861c-48c3-946b-5ffa93867689" containerName="revoke-keys" Oct 30 18:28:51.166515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.166217 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c29e32f-861c-48c3-946b-5ffa93867689" containerName="revoke-keys" Oct 30 18:28:51.166515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.166235 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" Oct 30 18:28:51.166515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.166243 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" Oct 30 18:28:51.166515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.166254 5624 cpu_manager.go:414] "containerMap: removing container" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerName="istio-proxy" Oct 30 18:28:51.166515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.166261 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerName="istio-proxy" Oct 30 18:28:51.179165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.179133 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerName="istio-proxy" Oct 30 18:28:51.180003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.179980 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c29e32f-861c-48c3-946b-5ffa93867689" containerName="revoke-keys" Oct 30 18:28:51.180147 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.180136 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" Oct 30 18:28:51.180235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.180224 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="2525bc9c-24dd-4ee0-afe9-2d79c0266d41" containerName="istio-proxy" Oct 30 18:28:51.477015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:51.407583 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:52.154095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.152865 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:52.169001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.154784 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:52.169001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.165996 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:28:52.169001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.166386 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.226729 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.222725 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1b0848-serving-cert\"" Oct 30 18:28:52.320957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.319635 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24"] Oct 30 18:28:52.336994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.322415 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.336994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.322493 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.336994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.326103 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbnx\" (UniqueName: \"kubernetes.io/projected/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-kube-api-access-xmbnx\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.336994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.326253 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-metrics-certs\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.428784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.428697 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.429048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.429026 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.438236 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.438206 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbnx\" (UniqueName: \"kubernetes.io/projected/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-kube-api-access-xmbnx\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.438514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.438499 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-metrics-certs\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.439148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.429678 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.453182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.453150 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-metrics-certs\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.472962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.468223 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.481846 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.481789 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbnx\" (UniqueName: \"kubernetes.io/projected/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-kube-api-access-xmbnx\") pod \"maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.586539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.586051 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:52.586539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.586127 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:52.641611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.641573 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:52.643248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.643185 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:52.808181 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.806370 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:52.940903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:52.940813 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:53.119407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.118829 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:53.181969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.173334 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:53.181969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.173866 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:28:53.181969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.174196 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" podUID="da2b19fd-2648-4e80-bde1-16211807bc16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:53.188283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.182840 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:28:53.216096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.216056 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:53.336330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.336019 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" podUID="2d81034f-ac92-46ff-a8d6-a432835b1426" containerName="payload-processing" containerID="cri-o://15892bb4ec4a425073be3198b8273086186c55d7246007db983e66248e0c6853" gracePeriod=25 Oct 30 18:28:53.345892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.345560 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:28:53.385167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.374145 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" podUID="8c0ef545-bf5b-4665-95a8-2c6b578138a3" containerName="payload-pre-processing" containerID="cri-o://1bbc97b10fc4aafadb02756054dcd8d507850108d79608ea1a48a36e48282935" gracePeriod=25 Oct 30 18:28:53.385167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.382730 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:53.463966 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.444035 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:53.595251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.576291 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:28:53.595251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:53.576381 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:54.062459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.062407 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd" Oct 30 18:28:54.126882 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.119394 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:54.126882 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.120066 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:28:54.259175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.259089 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24"] Oct 30 18:28:54.307979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.270674 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff"] Oct 30 18:28:54.307979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.286697 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht"] Oct 30 18:28:54.353456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.353370 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:54.357386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.357357 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:28:54.502884 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.502849 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:54.746347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.746316 5624 generic.go:356] "Generic (PLEG): container finished" podID="8c0ef545-bf5b-4665-95a8-2c6b578138a3" containerID="1bbc97b10fc4aafadb02756054dcd8d507850108d79608ea1a48a36e48282935" exitCode=0 Oct 30 18:28:54.746642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.746623 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" event={"ID":"8c0ef545-bf5b-4665-95a8-2c6b578138a3","Type":"ContainerDied","Data":"1bbc97b10fc4aafadb02756054dcd8d507850108d79608ea1a48a36e48282935"} Oct 30 18:28:54.749624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.749598 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" event={"ID":"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80","Type":"ContainerStarted","Data":"5575360b59b9d1cbe03ca8421b8af97c6b0e13748454dcc9e8a083cc4247f235"} Oct 30 18:28:54.753426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.753401 5624 generic.go:356] "Generic (PLEG): container finished" podID="2d81034f-ac92-46ff-a8d6-a432835b1426" containerID="15892bb4ec4a425073be3198b8273086186c55d7246007db983e66248e0c6853" exitCode=0 Oct 30 18:28:54.754566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.754522 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" event={"ID":"2d81034f-ac92-46ff-a8d6-a432835b1426","Type":"ContainerDied","Data":"15892bb4ec4a425073be3198b8273086186c55d7246007db983e66248e0c6853"} Oct 30 18:28:54.851602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.851574 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:28:54.869415 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.869388 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:54.945746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:54.945715 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:28:55.042971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.036554 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0ef545-bf5b-4665-95a8-2c6b578138a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0ef545-bf5b-4665-95a8-2c6b578138a3-plugins-config-volume\") pod \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\" (UID: \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\") " Oct 30 18:28:55.042971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.036693 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0ef545-bf5b-4665-95a8-2c6b578138a3-kube-api-access-8qmms\" (UniqueName: \"kubernetes.io/projected/8c0ef545-bf5b-4665-95a8-2c6b578138a3-kube-api-access-8qmms\") pod \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\" (UID: \"8c0ef545-bf5b-4665-95a8-2c6b578138a3\") " Oct 30 18:28:55.042971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.037664 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0ef545-bf5b-4665-95a8-2c6b578138a3-plugins-config-volume" pod "8c0ef545-bf5b-4665-95a8-2c6b578138a3" (UID: "8c0ef545-bf5b-4665-95a8-2c6b578138a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:55.141362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.138460 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d81034f-ac92-46ff-a8d6-a432835b1426-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d81034f-ac92-46ff-a8d6-a432835b1426-plugins-config-volume\") pod \"2d81034f-ac92-46ff-a8d6-a432835b1426\" (UID: \"2d81034f-ac92-46ff-a8d6-a432835b1426\") " Oct 30 18:28:55.141362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.138611 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d81034f-ac92-46ff-a8d6-a432835b1426-kube-api-access-pfhrl\" (UniqueName: \"kubernetes.io/projected/2d81034f-ac92-46ff-a8d6-a432835b1426-kube-api-access-pfhrl\") pod \"2d81034f-ac92-46ff-a8d6-a432835b1426\" (UID: \"2d81034f-ac92-46ff-a8d6-a432835b1426\") " Oct 30 18:28:55.141362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.139313 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c0ef545-bf5b-4665-95a8-2c6b578138a3-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:55.160957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.148453 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d81034f-ac92-46ff-a8d6-a432835b1426-plugins-config-volume" pod "2d81034f-ac92-46ff-a8d6-a432835b1426" (UID: "2d81034f-ac92-46ff-a8d6-a432835b1426"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:55.166531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.166481 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0ef545-bf5b-4665-95a8-2c6b578138a3-kube-api-access-8qmms" pod "8c0ef545-bf5b-4665-95a8-2c6b578138a3" (UID: "8c0ef545-bf5b-4665-95a8-2c6b578138a3"). InnerVolumeSpecName "kube-api-access-8qmms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:55.167359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.167318 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d81034f-ac92-46ff-a8d6-a432835b1426-kube-api-access-pfhrl" pod "2d81034f-ac92-46ff-a8d6-a432835b1426" (UID: "2d81034f-ac92-46ff-a8d6-a432835b1426"). InnerVolumeSpecName "kube-api-access-pfhrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:55.240731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.240698 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d81034f-ac92-46ff-a8d6-a432835b1426-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:55.240978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.240965 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfhrl\" (UniqueName: \"kubernetes.io/projected/2d81034f-ac92-46ff-a8d6-a432835b1426-kube-api-access-pfhrl\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:55.241050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.241042 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qmms\" (UniqueName: \"kubernetes.io/projected/8c0ef545-bf5b-4665-95a8-2c6b578138a3-kube-api-access-8qmms\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:28:55.762606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.762574 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" event={"ID":"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80","Type":"ContainerStarted","Data":"688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43"} Oct 30 18:28:55.763050 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.762677 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:28:55.764388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.764370 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" Oct 30 18:28:55.764508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.764414 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4" event={"ID":"2d81034f-ac92-46ff-a8d6-a432835b1426","Type":"ContainerDied","Data":"a2a05a15e0bc72daf2f8459dc13b2bea004ef455bcf0a3d21385ef2c4c93210d"} Oct 30 18:28:55.764508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.764467 5624 scope.go:122] "RemoveContainer" containerID="15892bb4ec4a425073be3198b8273086186c55d7246007db983e66248e0c6853" Oct 30 18:28:55.766129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.766101 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" event={"ID":"8c0ef545-bf5b-4665-95a8-2c6b578138a3","Type":"ContainerDied","Data":"560e5cad8e9ac723fc6c2514c8137e18400170c6b9e06efc41c3a86ed1af29cc"} Oct 30 18:28:55.766221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.766127 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5" Oct 30 18:28:55.780305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.780288 5624 scope.go:122] "RemoveContainer" containerID="1bbc97b10fc4aafadb02756054dcd8d507850108d79608ea1a48a36e48282935" Oct 30 18:28:55.828484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.828225 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" podStartSLOduration=4.2988394549999995 podStartE2EDuration="4.828206425s" totalImagesPullingTime="529.36697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:55.808332105 +0000 UTC m=+1630.196860222" watchObservedRunningTime="2026-10-30 18:28:55.828206425 +0000 UTC m=+1630.216734542" Oct 30 18:28:55.864482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.864450 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5"] Oct 30 18:28:55.869018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.868992 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b2b596-6565bc59bc-s2qk5"] Oct 30 18:28:55.943065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.941715 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4"] Oct 30 18:28:55.949135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:55.949027 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b2b596-6ddfd66b58-r5hp4"] Oct 30 18:28:56.522961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:56.515328 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:57.797076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:57.797038 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d81034f-ac92-46ff-a8d6-a432835b1426" path="/var/lib/kubelet/pods/2d81034f-ac92-46ff-a8d6-a432835b1426/volumes" Oct 30 18:28:57.797712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:57.797691 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0ef545-bf5b-4665-95a8-2c6b578138a3" path="/var/lib/kubelet/pods/8c0ef545-bf5b-4665-95a8-2c6b578138a3/volumes" Oct 30 18:28:58.157723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:58.157688 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:59.271285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:59.271233 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" podUID="f339d950-7963-41f3-99ea-237bca9464dd" containerName="payload-processing" containerID="cri-o://cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3" gracePeriod=25 Oct 30 18:28:59.287903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:59.287848 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" containerID="cri-o://3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7" gracePeriod=25 Oct 30 18:28:59.615682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:28:59.615599 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:00.183048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:00.183012 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:00.284701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:00.284642 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:00.394067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:00.394032 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:00.821055 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:00.821013 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:01.674192 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:01.674127 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:01.790249 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:01.790206 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:01.793995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:01.793962 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:29:02.069861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.069828 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr"] Oct 30 18:29:02.070167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.070134 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" containerID="cri-o://a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c" gracePeriod=30 Oct 30 18:29:02.091355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.091322 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:02.254917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.249367 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:02.928868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.928833 5624 generic.go:356] "Generic (PLEG): container finished" podID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerID="a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4" exitCode=0 Oct 30 18:29:02.929308 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.928909 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" event={"ID":"ac9bf78e-2113-4935-a722-92de3a6be1b1","Type":"ContainerDied","Data":"a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4"} Oct 30 18:29:02.929391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.929375 5624 scope.go:122] "RemoveContainer" containerID="a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4" Oct 30 18:29:02.931063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.931035 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:02.931197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.931103 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:02.931197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.931136 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:02.953377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.953352 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" containerID="cri-o://215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f" gracePeriod=45 Oct 30 18:29:02.957019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:02.956993 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" containerID="cri-o://88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad" gracePeriod=45 Oct 30 18:29:03.068630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.068599 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": dial tcp 10.134.0.91:8000: connect: connection refused" Oct 30 18:29:03.236635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.236383 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:03.289827 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.289795 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:29:03.378640 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.378605 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-dshm\") pod \"f0d814b1-9303-4d63-804f-0487021f295b\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " Oct 30 18:29:03.378795 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.378716 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-model-cache\") pod \"f0d814b1-9303-4d63-804f-0487021f295b\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " Oct 30 18:29:03.378902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.378886 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d814b1-9303-4d63-804f-0487021f295b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d814b1-9303-4d63-804f-0487021f295b-tls-certs\") pod \"f0d814b1-9303-4d63-804f-0487021f295b\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " Oct 30 18:29:03.378988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.378924 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-home\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-home\") pod \"f0d814b1-9303-4d63-804f-0487021f295b\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " Oct 30 18:29:03.379053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.378993 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d814b1-9303-4d63-804f-0487021f295b-kube-api-access-j2svk\" (UniqueName: \"kubernetes.io/projected/f0d814b1-9303-4d63-804f-0487021f295b-kube-api-access-j2svk\") pod \"f0d814b1-9303-4d63-804f-0487021f295b\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " Oct 30 18:29:03.379053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.379029 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-tmp-dir\") pod \"f0d814b1-9303-4d63-804f-0487021f295b\" (UID: \"f0d814b1-9303-4d63-804f-0487021f295b\") " Oct 30 18:29:03.379575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.379548 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-model-cache" pod "f0d814b1-9303-4d63-804f-0487021f295b" (UID: "f0d814b1-9303-4d63-804f-0487021f295b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.379794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.379777 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-home" pod "f0d814b1-9303-4d63-804f-0487021f295b" (UID: "f0d814b1-9303-4d63-804f-0487021f295b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.380292 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.380273 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-tmp-dir" pod "f0d814b1-9303-4d63-804f-0487021f295b" (UID: "f0d814b1-9303-4d63-804f-0487021f295b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.383402 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.383375 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-dshm" pod "f0d814b1-9303-4d63-804f-0487021f295b" (UID: "f0d814b1-9303-4d63-804f-0487021f295b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.390589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.390551 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d814b1-9303-4d63-804f-0487021f295b-tls-certs" pod "f0d814b1-9303-4d63-804f-0487021f295b" (UID: "f0d814b1-9303-4d63-804f-0487021f295b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:03.399887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.397901 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:29:03.400407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.400386 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d814b1-9303-4d63-804f-0487021f295b-kube-api-access-j2svk" pod "f0d814b1-9303-4d63-804f-0487021f295b" (UID: "f0d814b1-9303-4d63-804f-0487021f295b"). InnerVolumeSpecName "kube-api-access-j2svk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:03.480722 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.480645 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc1e591-0916-4cad-85a0-10091e148e43-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccc1e591-0916-4cad-85a0-10091e148e43-tls-certs\") pod \"ccc1e591-0916-4cad-85a0-10091e148e43\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " Oct 30 18:29:03.480888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.480815 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc1e591-0916-4cad-85a0-10091e148e43-kube-api-access-2pmqn\" (UniqueName: \"kubernetes.io/projected/ccc1e591-0916-4cad-85a0-10091e148e43-kube-api-access-2pmqn\") pod \"ccc1e591-0916-4cad-85a0-10091e148e43\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " Oct 30 18:29:03.480974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.480898 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-home\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-home\") pod \"ccc1e591-0916-4cad-85a0-10091e148e43\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " Oct 30 18:29:03.480974 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.480949 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-model-cache\") pod \"ccc1e591-0916-4cad-85a0-10091e148e43\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " Oct 30 18:29:03.481100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481038 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-tmp-dir\") pod \"ccc1e591-0916-4cad-85a0-10091e148e43\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " Oct 30 18:29:03.481100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481086 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-dshm\") pod \"ccc1e591-0916-4cad-85a0-10091e148e43\" (UID: \"ccc1e591-0916-4cad-85a0-10091e148e43\") " Oct 30 18:29:03.481322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481301 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-model-cache" pod "ccc1e591-0916-4cad-85a0-10091e148e43" (UID: "ccc1e591-0916-4cad-85a0-10091e148e43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481465 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-tmp-dir" pod "ccc1e591-0916-4cad-85a0-10091e148e43" (UID: "ccc1e591-0916-4cad-85a0-10091e148e43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481745 5624 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-dshm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481766 5624 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-model-cache\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481782 5624 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-model-cache\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481796 5624 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-tmp-dir\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481807 5624 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d814b1-9303-4d63-804f-0487021f295b-tls-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481820 5624 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-home\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481833 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2svk\" (UniqueName: \"kubernetes.io/projected/f0d814b1-9303-4d63-804f-0487021f295b-kube-api-access-j2svk\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.481880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.481847 5624 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d814b1-9303-4d63-804f-0487021f295b-tmp-dir\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.482589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.482562 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-home" pod "ccc1e591-0916-4cad-85a0-10091e148e43" (UID: "ccc1e591-0916-4cad-85a0-10091e148e43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.483842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.483814 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc1e591-0916-4cad-85a0-10091e148e43-tls-certs" pod "ccc1e591-0916-4cad-85a0-10091e148e43" (UID: "ccc1e591-0916-4cad-85a0-10091e148e43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:03.484697 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.484675 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc1e591-0916-4cad-85a0-10091e148e43-kube-api-access-2pmqn" pod "ccc1e591-0916-4cad-85a0-10091e148e43" (UID: "ccc1e591-0916-4cad-85a0-10091e148e43"). InnerVolumeSpecName "kube-api-access-2pmqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:03.485725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.485691 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-dshm" pod "ccc1e591-0916-4cad-85a0-10091e148e43" (UID: "ccc1e591-0916-4cad-85a0-10091e148e43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.592958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.589869 5624 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-home\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.592958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.589900 5624 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e591-0916-4cad-85a0-10091e148e43-dshm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.592958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.589912 5624 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccc1e591-0916-4cad-85a0-10091e148e43-tls-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.592958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.589928 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pmqn\" (UniqueName: \"kubernetes.io/projected/ccc1e591-0916-4cad-85a0-10091e148e43-kube-api-access-2pmqn\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.616462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.616360 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:03.749623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.749500 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:03.851638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.851607 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:03.940431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.940400 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" event={"ID":"ac9bf78e-2113-4935-a722-92de3a6be1b1","Type":"ContainerStarted","Data":"4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704"} Oct 30 18:29:03.941901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.941877 5624 generic.go:356] "Generic (PLEG): container finished" podID="f0d814b1-9303-4d63-804f-0487021f295b" containerID="215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f" exitCode=0 Oct 30 18:29:03.942086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.941925 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" Oct 30 18:29:03.942086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.941977 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" event={"ID":"f0d814b1-9303-4d63-804f-0487021f295b","Type":"ContainerDied","Data":"215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f"} Oct 30 18:29:03.942086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.942015 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2" event={"ID":"f0d814b1-9303-4d63-804f-0487021f295b","Type":"ContainerDied","Data":"5c8055f2da9aa3dd7dc4dc9fd2ed29a793b6135a0968f0902914874824b00cf7"} Oct 30 18:29:03.942086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.942033 5624 scope.go:122] "RemoveContainer" containerID="215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f" Oct 30 18:29:03.943497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.943473 5624 generic.go:356] "Generic (PLEG): container finished" podID="ccc1e591-0916-4cad-85a0-10091e148e43" containerID="88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad" exitCode=0 Oct 30 18:29:03.943620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.943506 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" Oct 30 18:29:03.943620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.943554 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" event={"ID":"ccc1e591-0916-4cad-85a0-10091e148e43","Type":"ContainerDied","Data":"88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad"} Oct 30 18:29:03.943620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.943585 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm" event={"ID":"ccc1e591-0916-4cad-85a0-10091e148e43","Type":"ContainerDied","Data":"497db115fe2ca6d927d1eac383b4a963df6dd244faad4d27c6be0e0d0c726255"} Oct 30 18:29:03.953744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.953724 5624 scope.go:122] "RemoveContainer" containerID="215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f" Oct 30 18:29:03.954016 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:03.953998 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f\": container with ID starting with 215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f not found: ID does not exist" containerID="215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f" Oct 30 18:29:03.954075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.954024 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f"} err="failed to get container status \"215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f\": rpc error: code = NotFound desc = could not find container \"215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f\": container with ID starting with 215ae14c504cd8dc9834cb90e1208cbe886b3a017102bb5585953d5c9c6ccd7f not found: ID does not exist" Oct 30 18:29:03.954075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.954040 5624 scope.go:122] "RemoveContainer" containerID="88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad" Oct 30 18:29:03.963429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.963390 5624 scope.go:122] "RemoveContainer" containerID="88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad" Oct 30 18:29:03.963697 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:03.963672 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad\": container with ID starting with 88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad not found: ID does not exist" containerID="88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad" Oct 30 18:29:03.963768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:03.963703 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad"} err="failed to get container status \"88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad\": rpc error: code = NotFound desc = could not find container \"88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad\": container with ID starting with 88872f3eebd08ff17d75d8f1cfc72fd4ea378cc7e4441ae304241548ce2991ad not found: ID does not exist" Oct 30 18:29:04.046316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:04.046237 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm"] Oct 30 18:29:04.050436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:04.050416 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/premium-w0-6ce5eb-kserve-778b9cddcb-wqnrm"] Oct 30 18:29:04.106960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:04.097991 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2"] Oct 30 18:29:04.118029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:04.117993 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/distinct-2-w0-6ce5eb-kserve-6455945f8d-pzlp2"] Oct 30 18:29:04.225795 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:04.225764 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:04.324034 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:04.323634 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:29:04.324034 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:04.323696 5624 patch_prober.go:29] interesting pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:29:04.324034 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:04.323748 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 18:29:05.315453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:05.315393 5624 patch_prober.go:29] interesting pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:29:05.316030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:05.315480 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 18:29:05.826051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:05.826013 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" path="/var/lib/kubelet/pods/ccc1e591-0916-4cad-85a0-10091e148e43/volumes" Oct 30 18:29:05.826563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:05.826546 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d814b1-9303-4d63-804f-0487021f295b" path="/var/lib/kubelet/pods/f0d814b1-9303-4d63-804f-0487021f295b/volumes" Oct 30 18:29:06.036961 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:06.036895 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:06.316726 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:06.316694 5624 patch_prober.go:29] interesting pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:29:06.317426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:06.317368 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 18:29:07.312352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.311850 5624 patch_prober.go:29] interesting pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:29:07.312352 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.311927 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 18:29:07.432958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.432678 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:29:07.476409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.476372 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:07.482100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482075 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d50df807-dc42-412b-8264-2ddc87c48176-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50df807-dc42-412b-8264-2ddc87c48176-istio-podinfo\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.482258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482117 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-certs\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.482258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482159 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-socket\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.482258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482237 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-credential-socket\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.482445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482310 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-kube-api-access-gjnnw\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-kube-api-access-gjnnw\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.482445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482340 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-istio-token\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-istio-token\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.482445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482413 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d50df807-dc42-412b-8264-2ddc87c48176-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50df807-dc42-412b-8264-2ddc87c48176-istiod-ca-cert\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.482602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482515 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-envoy\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.482602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482517 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-certs" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.482682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482589 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-socket" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.482925 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482706 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-credential-socket" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.482925 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482824 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50df807-dc42-412b-8264-2ddc87c48176-istiod-ca-cert" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:07.482925 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482826 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-data" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.483155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.482602 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-data\") pod \"d50df807-dc42-412b-8264-2ddc87c48176\" (UID: \"d50df807-dc42-412b-8264-2ddc87c48176\") " Oct 30 18:29:07.483519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.483500 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.483519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.483519 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50df807-dc42-412b-8264-2ddc87c48176-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.483658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.483529 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.483658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.483538 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.483658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.483545 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.508135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.506955 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-envoy" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.508135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.507100 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-istio-token" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:07.508135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.507460 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d50df807-dc42-412b-8264-2ddc87c48176-istio-podinfo" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:07.540970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.540909 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-kube-api-access-gjnnw" pod "d50df807-dc42-412b-8264-2ddc87c48176" (UID: "d50df807-dc42-412b-8264-2ddc87c48176"). InnerVolumeSpecName "kube-api-access-gjnnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:07.625667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.625570 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:07.625667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.625658 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50df807-dc42-412b-8264-2ddc87c48176-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.625899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.625682 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50df807-dc42-412b-8264-2ddc87c48176-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.625899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.625698 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjnnw\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-kube-api-access-gjnnw\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.625899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.625712 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50df807-dc42-412b-8264-2ddc87c48176-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.840877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.840802 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:07.861004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.860976 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk"] Oct 30 18:29:07.861440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.861422 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg"] Oct 30 18:29:07.862876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.861736 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" containerID="cri-o://8232a07492cde1cfa2442c3cec525f11c4c0b8dc528a7ef3ef2ea9fa7a33bb35" gracePeriod=30 Oct 30 18:29:07.863393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.863346 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" containerID="cri-o://845c1500826730553c30064c9c8218d6afcdef7fb149b8112c057b5d5179e596" gracePeriod=30 Oct 30 18:29:07.960520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:07.957874 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:08.116958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.116730 5624 generic.go:356] "Generic (PLEG): container finished" podID="d50df807-dc42-412b-8264-2ddc87c48176" containerID="a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c" exitCode=0 Oct 30 18:29:08.117158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.116971 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" event={"ID":"d50df807-dc42-412b-8264-2ddc87c48176","Type":"ContainerDied","Data":"a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c"} Oct 30 18:29:08.117158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.117045 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" event={"ID":"d50df807-dc42-412b-8264-2ddc87c48176","Type":"ContainerDied","Data":"ec62f99bc031888e3cb6453f4cd87485484d87fe8e7b036b2a63ada4d46dcec0"} Oct 30 18:29:08.117158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.117067 5624 scope.go:122] "RemoveContainer" containerID="a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c" Oct 30 18:29:08.117332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.117208 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr" Oct 30 18:29:08.128439 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.128422 5624 scope.go:122] "RemoveContainer" containerID="e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325" Oct 30 18:29:08.138505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.138483 5624 scope.go:122] "RemoveContainer" containerID="a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c" Oct 30 18:29:08.138844 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:08.138818 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c\": container with ID starting with a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c not found: ID does not exist" containerID="a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c" Oct 30 18:29:08.138960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.138852 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c"} err="failed to get container status \"a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c\": rpc error: code = NotFound desc = could not find container \"a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c\": container with ID starting with a3b1c3fe268395df623dbba99c9571196baae629a1db8733c8186bb387d78c3c not found: ID does not exist" Oct 30 18:29:08.138960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.138868 5624 scope.go:122] "RemoveContainer" containerID="e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325" Oct 30 18:29:08.139204 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:08.139184 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325\": container with ID starting with e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325 not found: ID does not exist" containerID="e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325" Oct 30 18:29:08.139204 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.139212 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325"} err="failed to get container status \"e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325\": rpc error: code = NotFound desc = could not find container \"e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325\": container with ID starting with e791fcc3b56ab38b4372e5da07845412d8dc4f9af2e2ae84d3fac49977758325 not found: ID does not exist" Oct 30 18:29:08.174986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.174965 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr"] Oct 30 18:29:08.180894 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.180871 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-h6rxr"] Oct 30 18:29:08.312638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.312599 5624 patch_prober.go:29] interesting pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:29:08.312814 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.312666 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 18:29:08.911441 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.910363 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:08.912281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:08.912250 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" podUID="f339d950-7963-41f3-99ea-237bca9464dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:09.312574 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:09.312483 5624 patch_prober.go:29] interesting pod/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 18:29:09.312574 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:09.312541 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 18:29:09.428265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:09.428219 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:09.792500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:09.792470 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d50df807-dc42-412b-8264-2ddc87c48176" path="/var/lib/kubelet/pods/d50df807-dc42-412b-8264-2ddc87c48176/volumes" Oct 30 18:29:10.267563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:10.267522 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:29:10.317702 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:10.317672 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:29:10.454486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:10.454455 5624 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:29:10.454654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:10.454503 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:29:11.021824 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:11.021787 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:11.141591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:11.141564 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:29:12.531057 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:12.531023 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:12.796848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:12.796758 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:13.162120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.162083 5624 generic.go:356] "Generic (PLEG): container finished" podID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerID="8232a07492cde1cfa2442c3cec525f11c4c0b8dc528a7ef3ef2ea9fa7a33bb35" exitCode=0 Oct 30 18:29:13.162310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.162139 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" event={"ID":"3b0177a2-63ed-4c8a-887a-100f3e9c02c3","Type":"ContainerDied","Data":"8232a07492cde1cfa2442c3cec525f11c4c0b8dc528a7ef3ef2ea9fa7a33bb35"} Oct 30 18:29:13.164763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.164735 5624 generic.go:356] "Generic (PLEG): container finished" podID="90b5fe30-d722-4430-96c1-14044e6450f0" containerID="845c1500826730553c30064c9c8218d6afcdef7fb149b8112c057b5d5179e596" exitCode=0 Oct 30 18:29:13.164908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.164806 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" event={"ID":"90b5fe30-d722-4430-96c1-14044e6450f0","Type":"ContainerDied","Data":"845c1500826730553c30064c9c8218d6afcdef7fb149b8112c057b5d5179e596"} Oct 30 18:29:13.230948 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.230916 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:29:13.257524 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.257497 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:29:13.263347 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263325 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-credential-socket\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263461 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263399 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-certs\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263461 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263440 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-envoy\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263488 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-data\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263519 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-socket\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263591 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-podinfo\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263633 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-credential-socket" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.263705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263656 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-kube-api-access-j92t7\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-kube-api-access-j92t7\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263699 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-token\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-token\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263768 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-data" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.263926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263791 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istiod-ca-cert\") pod \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\" (UID: \"3b0177a2-63ed-4c8a-887a-100f3e9c02c3\") " Oct 30 18:29:13.263926 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.263873 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-socket" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.264437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.264415 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.264514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.264442 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.264514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.264452 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.264622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.264549 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-certs" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.264851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.264824 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istiod-ca-cert" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:13.266077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.266056 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-envoy" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.266255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.266233 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-token" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.266316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.266279 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-podinfo" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:13.266364 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.266322 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-kube-api-access-j92t7" pod "3b0177a2-63ed-4c8a-887a-100f3e9c02c3" (UID: "3b0177a2-63ed-4c8a-887a-100f3e9c02c3"). InnerVolumeSpecName "kube-api-access-j92t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.365085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365006 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b5fe30-d722-4430-96c1-14044e6450f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b5fe30-d722-4430-96c1-14044e6450f0-istiod-ca-cert\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.365085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365054 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-socket\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.365310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365106 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-certs\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.365310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365166 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-envoy\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.365310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365194 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-credential-socket\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.365310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365228 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-data\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.365310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365272 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90b5fe30-d722-4430-96c1-14044e6450f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b5fe30-d722-4430-96c1-14044e6450f0-istio-podinfo\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.365310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365304 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-kube-api-access-5tn9x\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-kube-api-access-5tn9x\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.365825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365438 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b5fe30-d722-4430-96c1-14044e6450f0-istiod-ca-cert" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:13.365825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.365514 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-istio-token\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-istio-token\") pod \"90b5fe30-d722-4430-96c1-14044e6450f0\" (UID: \"90b5fe30-d722-4430-96c1-14044e6450f0\") " Oct 30 18:29:13.366063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366046 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366068 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366081 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b5fe30-d722-4430-96c1-14044e6450f0-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366093 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366105 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92t7\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-kube-api-access-j92t7\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366118 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.366151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366131 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0177a2-63ed-4c8a-887a-100f3e9c02c3-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.367002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366322 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-socket" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.367002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366579 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-certs" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.367002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.366849 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-credential-socket" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.367298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.367274 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-data" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.370267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.370243 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-envoy" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:13.370384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.370283 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-kube-api-access-5tn9x" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "kube-api-access-5tn9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.370384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.370359 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-istio-token" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:13.371624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.371602 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90b5fe30-d722-4430-96c1-14044e6450f0-istio-podinfo" pod "90b5fe30-d722-4430-96c1-14044e6450f0" (UID: "90b5fe30-d722-4430-96c1-14044e6450f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:13.467391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.467364 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.467391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.467392 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.467645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.467405 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.467645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.467419 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.467645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.467432 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.467645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.467444 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b5fe30-d722-4430-96c1-14044e6450f0-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.467645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.467456 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b5fe30-d722-4430-96c1-14044e6450f0-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.467645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.467468 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tn9x\" (UniqueName: \"kubernetes.io/projected/90b5fe30-d722-4430-96c1-14044e6450f0-kube-api-access-5tn9x\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:13.567812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.567773 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm"] Oct 30 18:29:13.608818 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.608785 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x"] Oct 30 18:29:13.720151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.720107 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:13.813845 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:13.813818 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:14.173331 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.173303 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" Oct 30 18:29:14.173525 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.173331 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk" event={"ID":"3b0177a2-63ed-4c8a-887a-100f3e9c02c3","Type":"ContainerDied","Data":"714d919a10cd16812fcace84e67a07c921e8e8853fdb342eaee078dd36b61e23"} Oct 30 18:29:14.173525 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.173396 5624 scope.go:122] "RemoveContainer" containerID="8232a07492cde1cfa2442c3cec525f11c4c0b8dc528a7ef3ef2ea9fa7a33bb35" Oct 30 18:29:14.175139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.175108 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" Oct 30 18:29:14.175274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.175107 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg" event={"ID":"90b5fe30-d722-4430-96c1-14044e6450f0","Type":"ContainerDied","Data":"6a4c8054069bb3b94f5fecaea39659fb9aba9e8504edd15d12dc626633f848c6"} Oct 30 18:29:14.184894 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.184868 5624 scope.go:122] "RemoveContainer" containerID="845c1500826730553c30064c9c8218d6afcdef7fb149b8112c057b5d5179e596" Oct 30 18:29:14.233981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.233953 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk"] Oct 30 18:29:14.244837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.244811 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-nl5rk"] Oct 30 18:29:14.284479 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.284444 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg"] Oct 30 18:29:14.288268 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.288244 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-ht6hg"] Oct 30 18:29:14.418223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:14.411754 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:15.832628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:15.832591 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" path="/var/lib/kubelet/pods/3b0177a2-63ed-4c8a-887a-100f3e9c02c3/volumes" Oct 30 18:29:15.833328 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:15.833307 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" path="/var/lib/kubelet/pods/90b5fe30-d722-4430-96c1-14044e6450f0/volumes" Oct 30 18:29:15.833742 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:15.833727 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2"] Oct 30 18:29:15.834045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:15.834008 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" podUID="c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" containerName="maas-api" containerID="cri-o://fd2344d3f2e79849a9bff032e82460446eb04ca6cf2132bb824cae4989d29ce4" gracePeriod=30 Oct 30 18:29:15.857879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:15.857852 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:15.963854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:15.963828 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:16.315957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.313300 5624 generic.go:356] "Generic (PLEG): container finished" podID="c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" containerID="fd2344d3f2e79849a9bff032e82460446eb04ca6cf2132bb824cae4989d29ce4" exitCode=0 Oct 30 18:29:16.315957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.313389 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" event={"ID":"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac","Type":"ContainerDied","Data":"fd2344d3f2e79849a9bff032e82460446eb04ca6cf2132bb824cae4989d29ce4"} Oct 30 18:29:16.404985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.403110 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:29:16.422256 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.421032 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:16.521281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.517420 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-openshift-service-ca\") pod \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " Oct 30 18:29:16.521281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.517484 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-maas-api-tls\") pod \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " Oct 30 18:29:16.521281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.517572 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-metrics-certs\") pod \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " Oct 30 18:29:16.521281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.517623 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-kube-api-access-zp2c8\" (UniqueName: \"kubernetes.io/projected/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-kube-api-access-zp2c8\") pod \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\" (UID: \"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac\") " Oct 30 18:29:16.521281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.518884 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-openshift-service-ca" pod "c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" (UID: "c56b6bd9-b1d9-483f-92f8-8a4aadad34ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:16.525232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.523432 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:16.525232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.525028 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-metrics-certs" pod "c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" (UID: "c56b6bd9-b1d9-483f-92f8-8a4aadad34ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:16.530631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.529009 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-maas-api-tls" pod "c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" (UID: "c56b6bd9-b1d9-483f-92f8-8a4aadad34ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:16.531177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.531149 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-kube-api-access-zp2c8" pod "c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" (UID: "c56b6bd9-b1d9-483f-92f8-8a4aadad34ac"). InnerVolumeSpecName "kube-api-access-zp2c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:16.637226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.635473 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:16.645139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.640915 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:16.645139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.645075 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:16.645139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.645093 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:16.645139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.645108 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp2c8\" (UniqueName: \"kubernetes.io/projected/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac-kube-api-access-zp2c8\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:16.957756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.957673 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4"] Oct 30 18:29:16.958339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958322 5624 cpu_manager.go:414] "containerMap: removing container" podUID="c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" containerName="maas-api" Oct 30 18:29:16.958483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958466 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" containerName="maas-api" Oct 30 18:29:16.958566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958490 5624 cpu_manager.go:414] "containerMap: removing container" podUID="2d81034f-ac92-46ff-a8d6-a432835b1426" containerName="payload-processing" Oct 30 18:29:16.958566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958498 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d81034f-ac92-46ff-a8d6-a432835b1426" containerName="payload-processing" Oct 30 18:29:16.958566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958513 5624 cpu_manager.go:414] "containerMap: removing container" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" Oct 30 18:29:16.958566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958520 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" Oct 30 18:29:16.958566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958531 5624 cpu_manager.go:414] "containerMap: removing container" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" Oct 30 18:29:16.958566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958539 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958571 5624 cpu_manager.go:414] "containerMap: removing container" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958578 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958588 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8c0ef545-bf5b-4665-95a8-2c6b578138a3" containerName="payload-pre-processing" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958596 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0ef545-bf5b-4665-95a8-2c6b578138a3" containerName="payload-pre-processing" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958615 5624 cpu_manager.go:414] "containerMap: removing container" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958622 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958633 5624 cpu_manager.go:414] "containerMap: removing container" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958640 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958655 5624 cpu_manager.go:414] "containerMap: removing container" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958662 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958710 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958718 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" Oct 30 18:29:16.958879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958877 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8626ce00-6a12-48e5-a761-c20623fd714a" containerName="istio-proxy" Oct 30 18:29:16.959559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958894 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" containerName="maas-api" Oct 30 18:29:16.959559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958903 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d81034f-ac92-46ff-a8d6-a432835b1426" containerName="payload-processing" Oct 30 18:29:16.959559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958913 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d814b1-9303-4d63-804f-0487021f295b" containerName="main" Oct 30 18:29:16.959559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958923 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" Oct 30 18:29:16.959559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958951 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0177a2-63ed-4c8a-887a-100f3e9c02c3" containerName="istio-proxy" Oct 30 18:29:16.959559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958960 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0ef545-bf5b-4665-95a8-2c6b578138a3" containerName="payload-pre-processing" Oct 30 18:29:16.959559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958973 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b5fe30-d722-4430-96c1-14044e6450f0" containerName="istio-proxy" Oct 30 18:29:16.959559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.958983 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc1e591-0916-4cad-85a0-10091e148e43" containerName="main" Oct 30 18:29:16.963027 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:16.963008 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.012957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.011060 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.012957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.011308 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c9a107-openshift-default-dockercfg-x2fvs\"" Oct 30 18:29:17.028531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.028504 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4"] Oct 30 18:29:17.050392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050364 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvwj\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-kube-api-access-xfvwj\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.050558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050404 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-token\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.050558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050437 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a49bec-3cdd-4178-be33-f8a0194365d0-istiod-ca-cert\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.050642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050581 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-data\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.050642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050624 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-socket\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.050766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050746 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-podinfo\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.050856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050836 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-credential-socket\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.050987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050973 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-envoy\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.051036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.050998 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-certs\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.151853 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-token\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.151912 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a49bec-3cdd-4178-be33-f8a0194365d0-istiod-ca-cert\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.151980 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-data\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.152025 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-socket\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.152085 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-podinfo\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.152136 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-credential-socket\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.152248 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-envoy\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.152280 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-certs\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.152340 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvwj\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-kube-api-access-xfvwj\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.153607 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a49bec-3cdd-4178-be33-f8a0194365d0-istiod-ca-cert\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.153963 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.153915 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-data\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.187212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.161552 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-credential-socket\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.187212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.161763 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-socket\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.187212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.162786 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-certs\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.187212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.175138 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.187212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.182290 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-podinfo\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.187212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.182810 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-envoy\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.221565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.221448 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-token\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.222460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.222407 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvwj\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-kube-api-access-xfvwj\") pod \"e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.286071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.285783 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:17.299968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.299876 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:17.357984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.355309 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" event={"ID":"c56b6bd9-b1d9-483f-92f8-8a4aadad34ac","Type":"ContainerDied","Data":"a2a51eab216277c95eb754d6d9565c5e78a6c240c7259e0f37ba53687cc0018d"} Oct 30 18:29:17.357984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.355373 5624 scope.go:122] "RemoveContainer" containerID="fd2344d3f2e79849a9bff032e82460446eb04ca6cf2132bb824cae4989d29ce4" Oct 30 18:29:17.357984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.355732 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2" Oct 30 18:29:17.455983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.455919 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2"] Oct 30 18:29:17.458499 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.458473 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6ce5eb-6c9f79f8ff-n26j2"] Oct 30 18:29:17.745813 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:29:17.745780 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a49bec_3cdd_4178_be33_f8a0194365d0.slice/crio-66ca6ff45c646dc03a863aa877828e454b7864f3116547c9f462cd8a7fa305e1 WatchSource:0}: Error finding container 66ca6ff45c646dc03a863aa877828e454b7864f3116547c9f462cd8a7fa305e1: Status 404 returned error can't find the container with id 66ca6ff45c646dc03a863aa877828e454b7864f3116547c9f462cd8a7fa305e1 Oct 30 18:29:17.746273 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.746250 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4"] Oct 30 18:29:17.748406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.748371 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:17.748508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.748451 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:17.748508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.748485 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:17.822204 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:17.822172 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56b6bd9-b1d9-483f-92f8-8a4aadad34ac" path="/var/lib/kubelet/pods/c56b6bd9-b1d9-483f-92f8-8a4aadad34ac/volumes" Oct 30 18:29:18.335310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:18.335281 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:18.365693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:18.365663 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" event={"ID":"c2a49bec-3cdd-4178-be33-f8a0194365d0","Type":"ContainerStarted","Data":"b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f"} Oct 30 18:29:18.365842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:18.365710 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" event={"ID":"c2a49bec-3cdd-4178-be33-f8a0194365d0","Type":"ContainerStarted","Data":"66ca6ff45c646dc03a863aa877828e454b7864f3116547c9f462cd8a7fa305e1"} Oct 30 18:29:18.440206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:18.440137 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" podStartSLOduration=2.440117317 podStartE2EDuration="2.440117317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:18.409637146 +0000 UTC m=+1652.798165264" watchObservedRunningTime="2026-10-30 18:29:18.440117317 +0000 UTC m=+1652.828645436" Oct 30 18:29:18.568174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:18.568130 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" podUID="8d01b1fb-3d3b-4831-8d03-ca892b7f93b5" containerName="payload-processing" containerID="cri-o://678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45" gracePeriod=25 Oct 30 18:29:18.618958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:18.609392 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" podUID="ee5e82a6-72ac-49e5-aebd-86fe992c0919" containerName="payload-pre-processing" containerID="cri-o://e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7" gracePeriod=25 Oct 30 18:29:18.931762 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:18.924902 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" podUID="f339d950-7963-41f3-99ea-237bca9464dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:18.931762 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:18.930747 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:19.044982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.041056 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:29:19.063626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.063600 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5e82a6-72ac-49e5-aebd-86fe992c0919-kube-api-access-rbzvq\" (UniqueName: \"kubernetes.io/projected/ee5e82a6-72ac-49e5-aebd-86fe992c0919-kube-api-access-rbzvq\") pod \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\" (UID: \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\") " Oct 30 18:29:19.063850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.063831 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5e82a6-72ac-49e5-aebd-86fe992c0919-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5e82a6-72ac-49e5-aebd-86fe992c0919-plugins-config-volume\") pod \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\" (UID: \"ee5e82a6-72ac-49e5-aebd-86fe992c0919\") " Oct 30 18:29:19.064265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.064242 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5e82a6-72ac-49e5-aebd-86fe992c0919-plugins-config-volume" pod "ee5e82a6-72ac-49e5-aebd-86fe992c0919" (UID: "ee5e82a6-72ac-49e5-aebd-86fe992c0919"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:19.064443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.064422 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5e82a6-72ac-49e5-aebd-86fe992c0919-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:19.066235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.066208 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5e82a6-72ac-49e5-aebd-86fe992c0919-kube-api-access-rbzvq" pod "ee5e82a6-72ac-49e5-aebd-86fe992c0919" (UID: "ee5e82a6-72ac-49e5-aebd-86fe992c0919"). InnerVolumeSpecName "kube-api-access-rbzvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:19.140821 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.140781 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:29:19.165136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.165106 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-plugins-config-volume\") pod \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\" (UID: \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\") " Oct 30 18:29:19.165300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.165193 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-kube-api-access-v9bqj\" (UniqueName: \"kubernetes.io/projected/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-kube-api-access-v9bqj\") pod \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\" (UID: \"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5\") " Oct 30 18:29:19.165744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.165721 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-plugins-config-volume" pod "8d01b1fb-3d3b-4831-8d03-ca892b7f93b5" (UID: "8d01b1fb-3d3b-4831-8d03-ca892b7f93b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:19.165897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.165727 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbzvq\" (UniqueName: \"kubernetes.io/projected/ee5e82a6-72ac-49e5-aebd-86fe992c0919-kube-api-access-rbzvq\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:19.167659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.167634 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-kube-api-access-v9bqj" pod "8d01b1fb-3d3b-4831-8d03-ca892b7f93b5" (UID: "8d01b1fb-3d3b-4831-8d03-ca892b7f93b5"). InnerVolumeSpecName "kube-api-access-v9bqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:19.267382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.267259 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:19.267382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.267310 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bqj\" (UniqueName: \"kubernetes.io/projected/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5-kube-api-access-v9bqj\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:19.300732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.294100 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:19.333186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.331564 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:19.377119 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.377084 5624 generic.go:356] "Generic (PLEG): container finished" podID="ee5e82a6-72ac-49e5-aebd-86fe992c0919" containerID="e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7" exitCode=0 Oct 30 18:29:19.377544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.377260 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" event={"ID":"ee5e82a6-72ac-49e5-aebd-86fe992c0919","Type":"ContainerDied","Data":"e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7"} Oct 30 18:29:19.377544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.377318 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" event={"ID":"ee5e82a6-72ac-49e5-aebd-86fe992c0919","Type":"ContainerDied","Data":"e56d7088fd7656406b711b4ce3a0138bf97573cef1dc0ddda980ee8a058139e9"} Oct 30 18:29:19.377544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.377353 5624 scope.go:122] "RemoveContainer" containerID="e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7" Oct 30 18:29:19.377544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.377498 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x" Oct 30 18:29:19.424114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.424077 5624 generic.go:356] "Generic (PLEG): container finished" podID="8d01b1fb-3d3b-4831-8d03-ca892b7f93b5" containerID="678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45" exitCode=0 Oct 30 18:29:19.425491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.425469 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" Oct 30 18:29:19.425599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.425486 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" event={"ID":"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5","Type":"ContainerDied","Data":"678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45"} Oct 30 18:29:19.425599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.425537 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm" event={"ID":"8d01b1fb-3d3b-4831-8d03-ca892b7f93b5","Type":"ContainerDied","Data":"b746a6f775c9899e1cd539ef75ece9ea77858c884481a1a4c4af18d3a1017d2f"} Oct 30 18:29:19.425906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.425738 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:19.426340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.426320 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:29:19.450105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.450083 5624 scope.go:122] "RemoveContainer" containerID="e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7" Oct 30 18:29:19.450442 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:19.450418 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7\": container with ID starting with e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7 not found: ID does not exist" containerID="e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7" Oct 30 18:29:19.450509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.450455 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7"} err="failed to get container status \"e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7\": rpc error: code = NotFound desc = could not find container \"e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7\": container with ID starting with e94acdb851fdcc13914283cd00a86ed2a99674f2f7e3fb9d7c334dee380843c7 not found: ID does not exist" Oct 30 18:29:19.450509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.450477 5624 scope.go:122] "RemoveContainer" containerID="678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45" Oct 30 18:29:19.465511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.465493 5624 scope.go:122] "RemoveContainer" containerID="678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45" Oct 30 18:29:19.465889 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:19.465861 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45\": container with ID starting with 678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45 not found: ID does not exist" containerID="678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45" Oct 30 18:29:19.466053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.465899 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45"} err="failed to get container status \"678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45\": rpc error: code = NotFound desc = could not find container \"678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45\": container with ID starting with 678fe0d3cb3a1e9d63f57a6ff0a3fb28e4ecbb80ec1c9aad8fd89138f3addd45 not found: ID does not exist" Oct 30 18:29:19.565305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.563068 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm"] Oct 30 18:29:19.570346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.569285 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-273105-5d95d79756-2tmtm"] Oct 30 18:29:19.651056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.651021 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x"] Oct 30 18:29:19.658044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.656910 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-273105-5bdf4b84dc-rpr9x"] Oct 30 18:29:19.746317 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.746088 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:29:19.776007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.775975 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c13883-2b83-47a6-a8c1-39ae5cc32976-kube-api-access-jx6k8\" (UniqueName: \"kubernetes.io/projected/69c13883-2b83-47a6-a8c1-39ae5cc32976-kube-api-access-jx6k8\") pod \"69c13883-2b83-47a6-a8c1-39ae5cc32976\" (UID: \"69c13883-2b83-47a6-a8c1-39ae5cc32976\") " Oct 30 18:29:19.776203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.776184 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c13883-2b83-47a6-a8c1-39ae5cc32976-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c13883-2b83-47a6-a8c1-39ae5cc32976-plugins-config-volume\") pod \"69c13883-2b83-47a6-a8c1-39ae5cc32976\" (UID: \"69c13883-2b83-47a6-a8c1-39ae5cc32976\") " Oct 30 18:29:19.776578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.776552 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c13883-2b83-47a6-a8c1-39ae5cc32976-plugins-config-volume" pod "69c13883-2b83-47a6-a8c1-39ae5cc32976" (UID: "69c13883-2b83-47a6-a8c1-39ae5cc32976"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:19.776725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.776704 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69c13883-2b83-47a6-a8c1-39ae5cc32976-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:19.778838 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.778815 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c13883-2b83-47a6-a8c1-39ae5cc32976-kube-api-access-jx6k8" pod "69c13883-2b83-47a6-a8c1-39ae5cc32976" (UID: "69c13883-2b83-47a6-a8c1-39ae5cc32976"). InnerVolumeSpecName "kube-api-access-jx6k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:19.812359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.812281 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:29:19.840616 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.840587 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d01b1fb-3d3b-4831-8d03-ca892b7f93b5" path="/var/lib/kubelet/pods/8d01b1fb-3d3b-4831-8d03-ca892b7f93b5/volumes" Oct 30 18:29:19.842211 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.842175 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5e82a6-72ac-49e5-aebd-86fe992c0919" path="/var/lib/kubelet/pods/ee5e82a6-72ac-49e5-aebd-86fe992c0919/volumes" Oct 30 18:29:19.877796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.877761 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f339d950-7963-41f3-99ea-237bca9464dd-kube-api-access-6rqbn\" (UniqueName: \"kubernetes.io/projected/f339d950-7963-41f3-99ea-237bca9464dd-kube-api-access-6rqbn\") pod \"f339d950-7963-41f3-99ea-237bca9464dd\" (UID: \"f339d950-7963-41f3-99ea-237bca9464dd\") " Oct 30 18:29:19.878003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.877885 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f339d950-7963-41f3-99ea-237bca9464dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f339d950-7963-41f3-99ea-237bca9464dd-plugins-config-volume\") pod \"f339d950-7963-41f3-99ea-237bca9464dd\" (UID: \"f339d950-7963-41f3-99ea-237bca9464dd\") " Oct 30 18:29:19.878331 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.878296 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f339d950-7963-41f3-99ea-237bca9464dd-plugins-config-volume" pod "f339d950-7963-41f3-99ea-237bca9464dd" (UID: "f339d950-7963-41f3-99ea-237bca9464dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:19.878851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.878825 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f339d950-7963-41f3-99ea-237bca9464dd-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:19.878990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.878853 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx6k8\" (UniqueName: \"kubernetes.io/projected/69c13883-2b83-47a6-a8c1-39ae5cc32976-kube-api-access-jx6k8\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:19.880392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.880364 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f339d950-7963-41f3-99ea-237bca9464dd-kube-api-access-6rqbn" pod "f339d950-7963-41f3-99ea-237bca9464dd" (UID: "f339d950-7963-41f3-99ea-237bca9464dd"). InnerVolumeSpecName "kube-api-access-6rqbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:19.926204 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:19.926171 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:29:19.980666 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:19.980638 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqbn\" (UniqueName: \"kubernetes.io/projected/f339d950-7963-41f3-99ea-237bca9464dd-kube-api-access-6rqbn\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:20.082006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.081904 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:20.437233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.437201 5624 generic.go:356] "Generic (PLEG): container finished" podID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerID="3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7" exitCode=0 Oct 30 18:29:20.437714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.437283 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" Oct 30 18:29:20.437714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.437287 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" event={"ID":"69c13883-2b83-47a6-a8c1-39ae5cc32976","Type":"ContainerDied","Data":"3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7"} Oct 30 18:29:20.437714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.437354 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht" event={"ID":"69c13883-2b83-47a6-a8c1-39ae5cc32976","Type":"ContainerDied","Data":"cf1b5bfb2c3a3ff89f093e13e4369b38c8bae50bbae63691281898cd857709ab"} Oct 30 18:29:20.437714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.437390 5624 scope.go:122] "RemoveContainer" containerID="3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7" Oct 30 18:29:20.439124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.439097 5624 generic.go:356] "Generic (PLEG): container finished" podID="f339d950-7963-41f3-99ea-237bca9464dd" containerID="cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3" exitCode=0 Oct 30 18:29:20.439303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.439276 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" event={"ID":"f339d950-7963-41f3-99ea-237bca9464dd","Type":"ContainerDied","Data":"cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3"} Oct 30 18:29:20.439431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.439330 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" event={"ID":"f339d950-7963-41f3-99ea-237bca9464dd","Type":"ContainerDied","Data":"278893e636aab10868fd388cb811f1349bf71c6e5431bb0d369eb57a833e8cf3"} Oct 30 18:29:20.439431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.439407 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff" Oct 30 18:29:20.449066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.449050 5624 scope.go:122] "RemoveContainer" containerID="3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7" Oct 30 18:29:20.449331 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:20.449312 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7\": container with ID starting with 3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7 not found: ID does not exist" containerID="3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7" Oct 30 18:29:20.449409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.449338 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7"} err="failed to get container status \"3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7\": rpc error: code = NotFound desc = could not find container \"3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7\": container with ID starting with 3fcff7991d719d0d7a47a6718f9703f3475869259fedda1056207734f666a4c7 not found: ID does not exist" Oct 30 18:29:20.449409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.449355 5624 scope.go:122] "RemoveContainer" containerID="cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3" Oct 30 18:29:20.459343 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.459324 5624 scope.go:122] "RemoveContainer" containerID="cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3" Oct 30 18:29:20.459622 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:20.459604 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3\": container with ID starting with cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3 not found: ID does not exist" containerID="cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3" Oct 30 18:29:20.459676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.459628 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3"} err="failed to get container status \"cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3\": rpc error: code = NotFound desc = could not find container \"cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3\": container with ID starting with cf58a944ebfd2b92866aaffce70324f24c4a60a7c522c3bc10ae386fa27934b3 not found: ID does not exist" Oct 30 18:29:20.494555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.494526 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht"] Oct 30 18:29:20.501862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.501834 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce5eb-55b49dc6ff-x2hht"] Oct 30 18:29:20.537559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.537533 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff"] Oct 30 18:29:20.545788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.545763 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce5eb-56b5fbd574-2j5ff"] Oct 30 18:29:20.893690 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:20.893190 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:21.004516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:21.004466 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:21.618089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:21.618056 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:29:21.618534 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:21.618174 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:29:21.793244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:21.793210 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" path="/var/lib/kubelet/pods/69c13883-2b83-47a6-a8c1-39ae5cc32976/volumes" Oct 30 18:29:21.793817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:21.793799 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f339d950-7963-41f3-99ea-237bca9464dd" path="/var/lib/kubelet/pods/f339d950-7963-41f3-99ea-237bca9464dd/volumes" Oct 30 18:29:21.830849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:21.830808 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.599781 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.599752 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.705373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.705341 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:22.792725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.792686 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79"] Oct 30 18:29:22.793593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793570 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8d01b1fb-3d3b-4831-8d03-ca892b7f93b5" containerName="payload-processing" Oct 30 18:29:22.793593 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793593 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d01b1fb-3d3b-4831-8d03-ca892b7f93b5" containerName="payload-processing" Oct 30 18:29:22.793770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793613 5624 cpu_manager.go:414] "containerMap: removing container" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" Oct 30 18:29:22.793770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793621 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" Oct 30 18:29:22.793770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793674 5624 cpu_manager.go:414] "containerMap: removing container" podUID="ee5e82a6-72ac-49e5-aebd-86fe992c0919" containerName="payload-pre-processing" Oct 30 18:29:22.793770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793682 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5e82a6-72ac-49e5-aebd-86fe992c0919" containerName="payload-pre-processing" Oct 30 18:29:22.793770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793700 5624 cpu_manager.go:414] "containerMap: removing container" podUID="f339d950-7963-41f3-99ea-237bca9464dd" containerName="payload-processing" Oct 30 18:29:22.793770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793707 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f339d950-7963-41f3-99ea-237bca9464dd" containerName="payload-processing" Oct 30 18:29:22.794091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793856 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5e82a6-72ac-49e5-aebd-86fe992c0919" containerName="payload-pre-processing" Oct 30 18:29:22.794091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793874 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="f339d950-7963-41f3-99ea-237bca9464dd" containerName="payload-processing" Oct 30 18:29:22.794091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793886 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="d50df807-dc42-412b-8264-2ddc87c48176" containerName="istio-proxy" Oct 30 18:29:22.794091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793898 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d01b1fb-3d3b-4831-8d03-ca892b7f93b5" containerName="payload-processing" Oct 30 18:29:22.794091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.793909 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c13883-2b83-47a6-a8c1-39ae5cc32976" containerName="payload-pre-processing" Oct 30 18:29:22.822572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.822239 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79"] Oct 30 18:29:22.822572 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.822366 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:22.839817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.839769 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c9a107\"" Oct 30 18:29:22.867258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.867188 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c9a107-dockercfg-rc8rt\"" Oct 30 18:29:22.939394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.939355 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79\" (UID: \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:22.939897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.939864 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdhh\" (UniqueName: \"kubernetes.io/projected/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-kube-api-access-5jdhh\") pod \"payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79\" (UID: \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:22.965536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:22.965506 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz"] Oct 30 18:29:23.016105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.014193 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz"] Oct 30 18:29:23.016105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.014483 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:23.045312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.044470 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdhh\" (UniqueName: \"kubernetes.io/projected/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-kube-api-access-5jdhh\") pod \"payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79\" (UID: \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:23.045312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.044616 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79\" (UID: \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:23.045312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.044672 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz\" (UID: \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:23.045312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.044737 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2h7x\" (UniqueName: \"kubernetes.io/projected/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-kube-api-access-m2h7x\") pod \"payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz\" (UID: \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:23.045967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.045886 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79\" (UID: \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:23.065139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.065106 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdhh\" (UniqueName: \"kubernetes.io/projected/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-kube-api-access-5jdhh\") pod \"payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79\" (UID: \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:23.146222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.146188 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz\" (UID: \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:23.146403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.146254 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2h7x\" (UniqueName: \"kubernetes.io/projected/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-kube-api-access-m2h7x\") pod \"payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz\" (UID: \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:23.146951 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.146911 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz\" (UID: \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:23.159837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.159816 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:23.166709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.166687 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2h7x\" (UniqueName: \"kubernetes.io/projected/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-kube-api-access-m2h7x\") pod \"payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz\" (UID: \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:23.363001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.362960 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:23.481707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.481677 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:23.496809 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.496779 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79"] Oct 30 18:29:23.706405 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:29:23.696252 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f1aa4a_30bb_4314_8fb1_7cc7f5dd9c72.slice/crio-e1952acc494fba6a5c64b0787023989b742794f2e298871efbc6f8f3ada2a170 WatchSource:0}: Error finding container e1952acc494fba6a5c64b0787023989b742794f2e298871efbc6f8f3ada2a170: Status 404 returned error can't find the container with id e1952acc494fba6a5c64b0787023989b742794f2e298871efbc6f8f3ada2a170 Oct 30 18:29:23.706405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:23.704024 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz"] Oct 30 18:29:24.086264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.086229 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:24.303805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.303759 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:24.404802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.404758 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:24.482732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.482692 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" event={"ID":"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72","Type":"ContainerStarted","Data":"f0e38528ec9596b56f86f25b047539c02623ab2427ffb8d50aa8dc621ad8f604"} Oct 30 18:29:24.482918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.482748 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" event={"ID":"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72","Type":"ContainerStarted","Data":"e1952acc494fba6a5c64b0787023989b742794f2e298871efbc6f8f3ada2a170"} Oct 30 18:29:24.483091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.483070 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:24.484718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.484696 5624 generic.go:356] "Generic (PLEG): container finished" podID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerID="f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce" exitCode=0 Oct 30 18:29:24.484853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.484771 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" event={"ID":"8cf67015-168c-4270-828d-b4e3ff14f47a","Type":"ContainerDied","Data":"f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce"} Oct 30 18:29:24.485127 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.485105 5624 scope.go:122] "RemoveContainer" containerID="f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce" Oct 30 18:29:24.486675 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.486645 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:24.486771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.486714 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:24.486771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.486745 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:29:24.486771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.486750 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" event={"ID":"48ddf336-41e7-4c70-9cd0-0aa2446ab35b","Type":"ContainerStarted","Data":"a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7"} Oct 30 18:29:24.487005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.486796 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" event={"ID":"48ddf336-41e7-4c70-9cd0-0aa2446ab35b","Type":"ContainerStarted","Data":"686f2bd2c8100854c423da84669b7b4b4c444b8b4222b8cbad468f79fd37a24c"} Oct 30 18:29:24.487163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.487146 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:24.614625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.614540 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" podStartSLOduration=2.614522726 podStartE2EDuration="2.614522726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:24.546476236 +0000 UTC m=+1658.935004354" watchObservedRunningTime="2026-10-30 18:29:24.614522726 +0000 UTC m=+1659.003050847" Oct 30 18:29:24.619324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.619280 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" podStartSLOduration=2.619265528 podStartE2EDuration="2.619265528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:24.614232594 +0000 UTC m=+1659.002760711" watchObservedRunningTime="2026-10-30 18:29:24.619265528 +0000 UTC m=+1659.007793645" Oct 30 18:29:24.721000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.715231 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f"] Oct 30 18:29:24.721000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.715455 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" containerID="cri-o://1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8" gracePeriod=30 Oct 30 18:29:24.729244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:24.725173 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:25.077441 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:25.076604 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:29:25.525405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:25.525363 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" event={"ID":"8cf67015-168c-4270-828d-b4e3ff14f47a","Type":"ContainerStarted","Data":"49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402"} Oct 30 18:29:25.580528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:25.580481 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:25.701983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:25.701602 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:26.257566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:26.256868 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:29:26.257566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:26.257264 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:29:26.257566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:26.257374 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 18:29:27.223076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.222233 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd"] Oct 30 18:29:27.223076 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.222511 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" podUID="a1085804-d290-4dfd-8b33-bbbbc3c28009" containerName="maas-api" containerID="cri-o://23f5bf2cca0fc069817af4d1bcda2e83d4817735587037f3f9987debdb89df8c" gracePeriod=30 Oct 30 18:29:27.257579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.257546 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:29:27.257715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.257600 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 18:29:27.275657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.275616 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:27.553916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.553647 5624 generic.go:356] "Generic (PLEG): container finished" podID="a1085804-d290-4dfd-8b33-bbbbc3c28009" containerID="23f5bf2cca0fc069817af4d1bcda2e83d4817735587037f3f9987debdb89df8c" exitCode=0 Oct 30 18:29:27.553916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.553729 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" event={"ID":"a1085804-d290-4dfd-8b33-bbbbc3c28009","Type":"ContainerDied","Data":"23f5bf2cca0fc069817af4d1bcda2e83d4817735587037f3f9987debdb89df8c"} Oct 30 18:29:27.665622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.665593 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:29:27.853009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.852123 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1085804-d290-4dfd-8b33-bbbbc3c28009-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1085804-d290-4dfd-8b33-bbbbc3c28009-openshift-service-ca\") pod \"a1085804-d290-4dfd-8b33-bbbbc3c28009\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " Oct 30 18:29:27.857860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.855377 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-maas-api-tls\") pod \"a1085804-d290-4dfd-8b33-bbbbc3c28009\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " Oct 30 18:29:27.857860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.855543 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1085804-d290-4dfd-8b33-bbbbc3c28009-kube-api-access-x72h9\" (UniqueName: \"kubernetes.io/projected/a1085804-d290-4dfd-8b33-bbbbc3c28009-kube-api-access-x72h9\") pod \"a1085804-d290-4dfd-8b33-bbbbc3c28009\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " Oct 30 18:29:27.857860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.855585 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-metrics-certs\") pod \"a1085804-d290-4dfd-8b33-bbbbc3c28009\" (UID: \"a1085804-d290-4dfd-8b33-bbbbc3c28009\") " Oct 30 18:29:27.862871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.862839 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-metrics-certs" pod "a1085804-d290-4dfd-8b33-bbbbc3c28009" (UID: "a1085804-d290-4dfd-8b33-bbbbc3c28009"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:27.863333 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.863309 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1085804-d290-4dfd-8b33-bbbbc3c28009-openshift-service-ca" pod "a1085804-d290-4dfd-8b33-bbbbc3c28009" (UID: "a1085804-d290-4dfd-8b33-bbbbc3c28009"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:27.867095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.867073 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-maas-api-tls" pod "a1085804-d290-4dfd-8b33-bbbbc3c28009" (UID: "a1085804-d290-4dfd-8b33-bbbbc3c28009"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:27.867996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.867964 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1085804-d290-4dfd-8b33-bbbbc3c28009-kube-api-access-x72h9" pod "a1085804-d290-4dfd-8b33-bbbbc3c28009" (UID: "a1085804-d290-4dfd-8b33-bbbbc3c28009"). InnerVolumeSpecName "kube-api-access-x72h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:27.912525 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.912398 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:27.959064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.959030 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:27.959064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.959067 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72h9\" (UniqueName: \"kubernetes.io/projected/a1085804-d290-4dfd-8b33-bbbbc3c28009-kube-api-access-x72h9\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:27.959332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.959084 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1085804-d290-4dfd-8b33-bbbbc3c28009-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:27.959332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:27.959100 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1085804-d290-4dfd-8b33-bbbbc3c28009-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:28.017771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.017283 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:28.120061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.119966 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:28.225960 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.225433 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:28.260004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.258353 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:29:28.260004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.258411 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 18:29:28.561982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.561926 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" Oct 30 18:29:28.562229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.562206 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd" event={"ID":"a1085804-d290-4dfd-8b33-bbbbc3c28009","Type":"ContainerDied","Data":"3bf84416fd7ee8c15f5bf783f1db38a1bf08c63f0ff5e2a5670a0ceee889cde0"} Oct 30 18:29:28.562383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.562367 5624 scope.go:122] "RemoveContainer" containerID="23f5bf2cca0fc069817af4d1bcda2e83d4817735587037f3f9987debdb89df8c" Oct 30 18:29:28.628861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.628828 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd"] Oct 30 18:29:28.639282 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.639253 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-273105-5cffbd76f8-mncgd"] Oct 30 18:29:28.903981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:28.903498 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:29.158784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:29.158702 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt"] Oct 30 18:29:29.257013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:29.256982 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:29:29.257186 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:29.257047 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 18:29:29.268018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:29.267989 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:29.815086 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:29.812709 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1085804-d290-4dfd-8b33-bbbbc3c28009" path="/var/lib/kubelet/pods/a1085804-d290-4dfd-8b33-bbbbc3c28009/volumes" Oct 30 18:29:30.058359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.058332 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:29:30.111469 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.111434 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f69b18c-b172-4310-a145-8385d76f1cbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69b18c-b172-4310-a145-8385d76f1cbd-istiod-ca-cert\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.111669 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.111554 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-kube-api-access-tpzrn\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-kube-api-access-tpzrn\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.111669 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.111634 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-socket\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.111802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.111768 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-credential-socket\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.114968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.113039 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-envoy\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.114968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.113098 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-data\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.114968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.113153 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-token\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-token\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.114968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.113268 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-certs\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.114968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.113349 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-podinfo\") pod \"9f69b18c-b172-4310-a145-8385d76f1cbd\" (UID: \"9f69b18c-b172-4310-a145-8385d76f1cbd\") " Oct 30 18:29:30.114968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.114048 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-credential-socket" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.114968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.114265 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-socket" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.115868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.115269 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-data" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.116374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.116000 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f69b18c-b172-4310-a145-8385d76f1cbd-istiod-ca-cert" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:30.118001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.117972 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-certs" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.118241 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.118196 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-token" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:30.119660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.119642 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.119809 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.119794 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.119967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.119950 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.120097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.120085 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.120214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.120202 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.120330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.120318 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f69b18c-b172-4310-a145-8385d76f1cbd-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.120993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.120920 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-envoy" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:30.122529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.122483 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-kube-api-access-tpzrn" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "kube-api-access-tpzrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:30.122788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.122751 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-podinfo" pod "9f69b18c-b172-4310-a145-8385d76f1cbd" (UID: "9f69b18c-b172-4310-a145-8385d76f1cbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:30.207797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.206788 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:30.221261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.221203 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpzrn\" (UniqueName: \"kubernetes.io/projected/9f69b18c-b172-4310-a145-8385d76f1cbd-kube-api-access-tpzrn\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.221261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.221231 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.221261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.221241 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f69b18c-b172-4310-a145-8385d76f1cbd-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:30.263972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.261362 5624 patch_prober.go:29] interesting pod/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 18:29:30.263972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.261421 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 18:29:30.602958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.593442 5624 generic.go:356] "Generic (PLEG): container finished" podID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerID="1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8" exitCode=0 Oct 30 18:29:30.602958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.593677 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" event={"ID":"9f69b18c-b172-4310-a145-8385d76f1cbd","Type":"ContainerDied","Data":"1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8"} Oct 30 18:29:30.602958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.593725 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" event={"ID":"9f69b18c-b172-4310-a145-8385d76f1cbd","Type":"ContainerDied","Data":"082dc991ef36c5ffba284d5ffbf98740270e002c5e0e9d93c6343d48b72b2631"} Oct 30 18:29:30.602958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.593755 5624 scope.go:122] "RemoveContainer" containerID="1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8" Oct 30 18:29:30.602958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.593860 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f" Oct 30 18:29:30.639551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.639528 5624 scope.go:122] "RemoveContainer" containerID="1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8" Oct 30 18:29:30.640003 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:30.639977 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8\": container with ID starting with 1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8 not found: ID does not exist" containerID="1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8" Oct 30 18:29:30.640085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.640018 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8"} err="failed to get container status \"1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8\": rpc error: code = NotFound desc = could not find container \"1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8\": container with ID starting with 1f192ad3e00e55433b0f476ac6ce8458de5f5cdd5e7334f402c7b82ec948aee8 not found: ID does not exist" Oct 30 18:29:30.668845 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.668816 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f"] Oct 30 18:29:30.682298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:30.682270 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-wfn7f"] Oct 30 18:29:31.261796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:31.261764 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:29:31.262203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:31.262183 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:29:31.262848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:31.262827 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:29:31.791733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:31.791697 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" path="/var/lib/kubelet/pods/9f69b18c-b172-4310-a145-8385d76f1cbd/volumes" Oct 30 18:29:31.968598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:31.968563 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:33.553471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:33.553438 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:34.337072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:34.337037 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:34.919188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:34.919141 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:35.019628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:35.019593 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:35.170945 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:35.170846 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:35.284007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:35.283906 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld"] Oct 30 18:29:35.284206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:35.284172 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" containerID="cri-o://49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402" gracePeriod=30 Oct 30 18:29:35.297573 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:35.297433 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:35.545017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:35.536800 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:29:35.545017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:35.538123 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:29:35.786092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:35.786034 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:36.939819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:36.939776 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:38.452071 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.452036 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5"] Oct 30 18:29:38.452602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.452589 5624 cpu_manager.go:414] "containerMap: removing container" podUID="a1085804-d290-4dfd-8b33-bbbbc3c28009" containerName="maas-api" Oct 30 18:29:38.452711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.452602 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1085804-d290-4dfd-8b33-bbbbc3c28009" containerName="maas-api" Oct 30 18:29:38.452711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.452652 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" Oct 30 18:29:38.452711 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.452657 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" Oct 30 18:29:38.452806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.452738 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f69b18c-b172-4310-a145-8385d76f1cbd" containerName="istio-proxy" Oct 30 18:29:38.452806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.452750 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1085804-d290-4dfd-8b33-bbbbc3c28009" containerName="maas-api" Oct 30 18:29:38.456255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.456223 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.471681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.471657 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c9a107-serving-cert\"" Oct 30 18:29:38.490274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.490239 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5"] Oct 30 18:29:38.525721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.525688 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-maas-api-tls\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.526125 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.526099 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42rb\" (UniqueName: \"kubernetes.io/projected/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-kube-api-access-h42rb\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.526250 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.526146 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-metrics-certs\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.526250 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.526237 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.542491 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.542449 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:38.627729 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.627702 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42rb\" (UniqueName: \"kubernetes.io/projected/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-kube-api-access-h42rb\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.627849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.627738 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-metrics-certs\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.627849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.627801 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.627849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.627826 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-maas-api-tls\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.628426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.628404 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.630505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.630463 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-metrics-certs\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.630505 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.630489 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-maas-api-tls\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.630651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.630489 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:38.648477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.648448 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42rb\" (UniqueName: \"kubernetes.io/projected/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-kube-api-access-h42rb\") pod \"maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:38.733338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.733194 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:38.774163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:38.774126 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:39.080092 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:29:39.080059 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cc05eb_a93b_4bd2_90ce_9e7846a74319.slice/crio-ce61fd55e1429d0dd5fbd2cff6e27c5527e39c284d00fa41a13f021175417a09 WatchSource:0}: Error finding container ce61fd55e1429d0dd5fbd2cff6e27c5527e39c284d00fa41a13f021175417a09: Status 404 returned error can't find the container with id ce61fd55e1429d0dd5fbd2cff6e27c5527e39c284d00fa41a13f021175417a09 Oct 30 18:29:39.081862 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:39.081833 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5"] Oct 30 18:29:39.681297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:39.681212 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" event={"ID":"b7cc05eb-a93b-4bd2-90ce-9e7846a74319","Type":"ContainerStarted","Data":"c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6"} Oct 30 18:29:39.681297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:39.681272 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" event={"ID":"b7cc05eb-a93b-4bd2-90ce-9e7846a74319","Type":"ContainerStarted","Data":"ce61fd55e1429d0dd5fbd2cff6e27c5527e39c284d00fa41a13f021175417a09"} Oct 30 18:29:39.681790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:39.681393 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:39.763540 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:39.763437 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" podStartSLOduration=1.4313020779999999 podStartE2EDuration="1.76341986s" totalImagesPullingTime="332.117782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:39.762238582 +0000 UTC m=+1674.150766700" watchObservedRunningTime="2026-10-30 18:29:39.76341986 +0000 UTC m=+1674.151947979" Oct 30 18:29:40.499671 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.499622 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:40.579255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.579215 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:29:40.656353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656322 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf67015-168c-4270-828d-b4e3ff14f47a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf67015-168c-4270-828d-b4e3ff14f47a-istiod-ca-cert\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.656523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656386 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-certs\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.656523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656474 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-envoy\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.656656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656527 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-socket\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.656656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656554 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-podinfo\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.656656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656640 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-credential-socket\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.656831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656753 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-kube-api-access-f85fg\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-kube-api-access-f85fg\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.656831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656804 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-data\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.657030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656819 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-certs" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:40.657030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656847 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-token\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-token\") pod \"8cf67015-168c-4270-828d-b4e3ff14f47a\" (UID: \"8cf67015-168c-4270-828d-b4e3ff14f47a\") " Oct 30 18:29:40.657030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656855 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf67015-168c-4270-828d-b4e3ff14f47a-istiod-ca-cert" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:40.657030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.656961 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-socket" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:40.657234 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.657178 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-credential-socket" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:40.657537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.657497 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-data" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:40.657657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.657614 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.657657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.657647 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.657785 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.657661 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cf67015-168c-4270-828d-b4e3ff14f47a-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.657785 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.657675 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.659312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.659281 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-token" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:40.659448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.659369 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-podinfo" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:40.659709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.659682 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-envoy" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:40.659709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.659702 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-kube-api-access-f85fg" pod "8cf67015-168c-4270-828d-b4e3ff14f47a" (UID: "8cf67015-168c-4270-828d-b4e3ff14f47a"). InnerVolumeSpecName "kube-api-access-f85fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:40.702369 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.702331 5624 generic.go:356] "Generic (PLEG): container finished" podID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerID="49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402" exitCode=0 Oct 30 18:29:40.702780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.702419 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" event={"ID":"8cf67015-168c-4270-828d-b4e3ff14f47a","Type":"ContainerDied","Data":"49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402"} Oct 30 18:29:40.702780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.702469 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" Oct 30 18:29:40.702860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.702818 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld" event={"ID":"8cf67015-168c-4270-828d-b4e3ff14f47a","Type":"ContainerDied","Data":"d0880042b87e2cddbead6cca838e9555e8868772569a8991dc0ccd52c6b63921"} Oct 30 18:29:40.702919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.702860 5624 scope.go:122] "RemoveContainer" containerID="49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402" Oct 30 18:29:40.715756 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.715732 5624 scope.go:122] "RemoveContainer" containerID="f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce" Oct 30 18:29:40.726761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.726739 5624 scope.go:122] "RemoveContainer" containerID="49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402" Oct 30 18:29:40.727095 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:40.727072 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402\": container with ID starting with 49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402 not found: ID does not exist" containerID="49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402" Oct 30 18:29:40.727194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.727101 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402"} err="failed to get container status \"49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402\": rpc error: code = NotFound desc = could not find container \"49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402\": container with ID starting with 49a548eb7298da8ef52488de1e69617f8ad420f9cd94d5eeeb78333fccb0c402 not found: ID does not exist" Oct 30 18:29:40.727194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.727118 5624 scope.go:122] "RemoveContainer" containerID="f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce" Oct 30 18:29:40.727407 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:40.727389 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce\": container with ID starting with f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce not found: ID does not exist" containerID="f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce" Oct 30 18:29:40.727468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.727416 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce"} err="failed to get container status \"f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce\": rpc error: code = NotFound desc = could not find container \"f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce\": container with ID starting with f558286c39412a4f4271840239df0f9d67dcff81397c03b16c62e86ceaf705ce not found: ID does not exist" Oct 30 18:29:40.759073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.759050 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.759073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.759072 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f85fg\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-kube-api-access-f85fg\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.759321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.759083 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.759321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.759093 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.759321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.759101 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cf67015-168c-4270-828d-b4e3ff14f47a-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:40.766379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.766360 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld"] Oct 30 18:29:40.778477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:40.778456 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-273105-openshift-default-6d57bbcd9f-nslld"] Oct 30 18:29:41.562038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:41.562009 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:41.791343 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:41.791311 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" path="/var/lib/kubelet/pods/8cf67015-168c-4270-828d-b4e3ff14f47a/volumes" Oct 30 18:29:42.367749 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:42.367704 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:42.468659 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:42.468622 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:44.063005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.062974 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:44.185785 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.185742 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" containerID="cri-o://71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f" gracePeriod=45 Oct 30 18:29:44.469939 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.469907 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:29:44.607045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607013 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tmp-dir\") pod \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " Oct 30 18:29:44.607253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607079 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-dshm\") pod \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " Oct 30 18:29:44.607253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607114 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tls-certs\") pod \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " Oct 30 18:29:44.607253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607180 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-model-cache\") pod \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " Oct 30 18:29:44.607435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607288 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tmp-dir" pod "6a99c9dc-0e26-42a4-8d0f-105b71066bc2" (UID: "6a99c9dc-0e26-42a4-8d0f-105b71066bc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.607435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607330 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-home\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-home\") pod \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " Oct 30 18:29:44.607435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607361 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-kube-api-access-l7c7d\" (UniqueName: \"kubernetes.io/projected/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-kube-api-access-l7c7d\") pod \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\" (UID: \"6a99c9dc-0e26-42a4-8d0f-105b71066bc2\") " Oct 30 18:29:44.607586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607513 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-model-cache" pod "6a99c9dc-0e26-42a4-8d0f-105b71066bc2" (UID: "6a99c9dc-0e26-42a4-8d0f-105b71066bc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.607586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607572 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-home" pod "6a99c9dc-0e26-42a4-8d0f-105b71066bc2" (UID: "6a99c9dc-0e26-42a4-8d0f-105b71066bc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.607902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607882 5624 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tmp-dir\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.607902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607902 5624 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-model-cache\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.608129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.607915 5624 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-home\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.609393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.609372 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-dshm" pod "6a99c9dc-0e26-42a4-8d0f-105b71066bc2" (UID: "6a99c9dc-0e26-42a4-8d0f-105b71066bc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.609729 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.609714 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tls-certs" pod "6a99c9dc-0e26-42a4-8d0f-105b71066bc2" (UID: "6a99c9dc-0e26-42a4-8d0f-105b71066bc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:44.610026 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.610005 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-kube-api-access-l7c7d" pod "6a99c9dc-0e26-42a4-8d0f-105b71066bc2" (UID: "6a99c9dc-0e26-42a4-8d0f-105b71066bc2"). InnerVolumeSpecName "kube-api-access-l7c7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:44.708721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.708689 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7c7d\" (UniqueName: \"kubernetes.io/projected/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-kube-api-access-l7c7d\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.708721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.708716 5624 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-dshm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.708721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.708725 5624 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a99c9dc-0e26-42a4-8d0f-105b71066bc2-tls-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.736276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.736244 5624 generic.go:356] "Generic (PLEG): container finished" podID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerID="71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f" exitCode=0 Oct 30 18:29:44.736432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.736286 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" Oct 30 18:29:44.736432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.736325 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" event={"ID":"6a99c9dc-0e26-42a4-8d0f-105b71066bc2","Type":"ContainerDied","Data":"71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f"} Oct 30 18:29:44.736432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.736376 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" event={"ID":"6a99c9dc-0e26-42a4-8d0f-105b71066bc2","Type":"ContainerDied","Data":"c88f1e0e8db5f0558ab173b338c7bf2dce87d8766b74f65513a46748a5e25a51"} Oct 30 18:29:44.736432 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.736393 5624 scope.go:122] "RemoveContainer" containerID="71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f" Oct 30 18:29:44.747864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.747843 5624 scope.go:122] "RemoveContainer" containerID="71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f" Oct 30 18:29:44.748178 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:44.748158 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f\": container with ID starting with 71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f not found: ID does not exist" containerID="71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f" Oct 30 18:29:44.748262 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.748191 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f"} err="failed to get container status \"71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f\": rpc error: code = NotFound desc = could not find container \"71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f\": container with ID starting with 71d4f7dbcbaff0c38250bad93286ee89517f5c88f2c93659642423c057f1fa7f not found: ID does not exist" Oct 30 18:29:44.792042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.792015 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt"] Oct 30 18:29:44.804302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:44.804277 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt"] Oct 30 18:29:45.407121 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:45.407064 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-w1-fb2fa0-kserve-56f5d686bf-8gcdt" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:29:45.642682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:45.642653 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:45.709978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:45.709856 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:29:45.803576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:45.795738 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" path="/var/lib/kubelet/pods/6a99c9dc-0e26-42a4-8d0f-105b71066bc2/volumes" Oct 30 18:29:46.553364 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:46.553332 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt"] Oct 30 18:29:46.669390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:46.669353 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:46.784403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:46.784346 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:47.494658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:47.494583 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:49.034272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:49.034219 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:49.919646 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:29:49.919606 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:29:50.533658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:50.533623 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:51.554566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:51.554509 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" containerID="cri-o://fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1" gracePeriod=25 Oct 30 18:29:52.149645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:52.149606 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:53.878051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:53.877976 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:55.467846 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:55.467816 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:55.811732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:55.811639 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:56.555453 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:56.555396 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:56.655778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:56.655728 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:57.206503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:57.206430 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:58.890297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:29:58.890264 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:00.170679 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.170643 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt"] Oct 30 18:30:00.180296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.180026 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" Oct 30 18:30:00.180296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.180054 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" Oct 30 18:30:00.180296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.180082 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" Oct 30 18:30:00.180296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.180090 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" Oct 30 18:30:00.180296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.180177 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" Oct 30 18:30:00.180296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.180185 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" Oct 30 18:30:00.180771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.180478 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a99c9dc-0e26-42a4-8d0f-105b71066bc2" containerName="main" Oct 30 18:30:00.180771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.180499 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" Oct 30 18:30:00.193997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.193972 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:00.195244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.195220 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4"] Oct 30 18:30:00.197948 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.197909 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf67015-168c-4270-828d-b4e3ff14f47a" containerName="istio-proxy" Oct 30 18:30:00.206149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.204855 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:00.206458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.206442 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4"] Oct 30 18:30:00.214160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.214134 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt"] Oct 30 18:30:00.246315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.246240 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:30:00.259000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.257883 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z"] Oct 30 18:30:00.286262 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.286107 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:00.287975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.287952 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr"] Oct 30 18:30:00.296323 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.296299 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z"] Oct 30 18:30:00.296521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.296404 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:00.322061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.322030 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr"] Oct 30 18:30:00.322834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.322808 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4jz\" (UniqueName: \"kubernetes.io/projected/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-kube-api-access-gx4jz\") pod \"maas-api-key-cleanup-29889750-9cqqt\" (UID: \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:00.322905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.322889 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-9cqqt\" (UID: \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:00.322959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.322928 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febc95e-db96-4150-824e-c0838cce1ec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4\" (UID: \"4febc95e-db96-4150-824e-c0838cce1ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:00.323061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.323047 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgmr\" (UniqueName: \"kubernetes.io/projected/4febc95e-db96-4150-824e-c0838cce1ec7-kube-api-access-2cgmr\") pod \"maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4\" (UID: \"4febc95e-db96-4150-824e-c0838cce1ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:00.424871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.424783 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-9cqqt\" (UID: \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:00.424871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.424835 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febc95e-db96-4150-824e-c0838cce1ec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4\" (UID: \"4febc95e-db96-4150-824e-c0838cce1ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:00.425120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.425024 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgmr\" (UniqueName: \"kubernetes.io/projected/4febc95e-db96-4150-824e-c0838cce1ec7-kube-api-access-2cgmr\") pod \"maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4\" (UID: \"4febc95e-db96-4150-824e-c0838cce1ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:00.425120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.425071 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bgg\" (UniqueName: \"kubernetes.io/projected/676dcb93-cf50-4156-b4d9-998c3a90cd1a-kube-api-access-d2bgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr\" (UID: \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:00.425235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.425142 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16b7126-6234-4edf-bcf9-ee14e001720f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z\" (UID: \"c16b7126-6234-4edf-bcf9-ee14e001720f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:00.425309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.425280 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/c16b7126-6234-4edf-bcf9-ee14e001720f-kube-api-access-bq78m\") pod \"maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z\" (UID: \"c16b7126-6234-4edf-bcf9-ee14e001720f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:00.425372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.425332 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676dcb93-cf50-4156-b4d9-998c3a90cd1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr\" (UID: \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:00.425426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.425372 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4jz\" (UniqueName: \"kubernetes.io/projected/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-kube-api-access-gx4jz\") pod \"maas-api-key-cleanup-29889750-9cqqt\" (UID: \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:00.425706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.425684 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889750-9cqqt\" (UID: \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:00.426041 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.426023 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febc95e-db96-4150-824e-c0838cce1ec7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4\" (UID: \"4febc95e-db96-4150-824e-c0838cce1ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:00.455727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.455701 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgmr\" (UniqueName: \"kubernetes.io/projected/4febc95e-db96-4150-824e-c0838cce1ec7-kube-api-access-2cgmr\") pod \"maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4\" (UID: \"4febc95e-db96-4150-824e-c0838cce1ec7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:00.456482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.456460 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4jz\" (UniqueName: \"kubernetes.io/projected/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-kube-api-access-gx4jz\") pod \"maas-api-key-cleanup-29889750-9cqqt\" (UID: \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:00.526555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.526522 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bgg\" (UniqueName: \"kubernetes.io/projected/676dcb93-cf50-4156-b4d9-998c3a90cd1a-kube-api-access-d2bgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr\" (UID: \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:00.526752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.526685 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16b7126-6234-4edf-bcf9-ee14e001720f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z\" (UID: \"c16b7126-6234-4edf-bcf9-ee14e001720f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:00.526837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.526774 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/c16b7126-6234-4edf-bcf9-ee14e001720f-kube-api-access-bq78m\") pod \"maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z\" (UID: \"c16b7126-6234-4edf-bcf9-ee14e001720f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:00.526837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.526811 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676dcb93-cf50-4156-b4d9-998c3a90cd1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr\" (UID: \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:00.527747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.527714 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16b7126-6234-4edf-bcf9-ee14e001720f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z\" (UID: \"c16b7126-6234-4edf-bcf9-ee14e001720f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:00.527747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.527732 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676dcb93-cf50-4156-b4d9-998c3a90cd1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr\" (UID: \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:00.548093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.548060 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bgg\" (UniqueName: \"kubernetes.io/projected/676dcb93-cf50-4156-b4d9-998c3a90cd1a-kube-api-access-d2bgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr\" (UID: \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:00.549271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.549244 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/c16b7126-6234-4edf-bcf9-ee14e001720f-kube-api-access-bq78m\") pod \"maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z\" (UID: \"c16b7126-6234-4edf-bcf9-ee14e001720f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:00.582607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.582584 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:00.595053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.595020 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:00.636641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.636615 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:00.646806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.646754 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:00.718991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.718913 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:00.922774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.922745 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt"] Oct 30 18:30:00.965318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:00.965291 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4"] Oct 30 18:30:00.965509 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:30:00.965490 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4febc95e_db96_4150_824e_c0838cce1ec7.slice/crio-0b074eb8095924bf30e082f30d1c6317de7cb9e109c7e0e0276f4b296da2a692 WatchSource:0}: Error finding container 0b074eb8095924bf30e082f30d1c6317de7cb9e109c7e0e0276f4b296da2a692: Status 404 returned error can't find the container with id 0b074eb8095924bf30e082f30d1c6317de7cb9e109c7e0e0276f4b296da2a692 Oct 30 18:30:01.034161 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:30:01.034129 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16b7126_6234_4edf_bcf9_ee14e001720f.slice/crio-23df21d8c7bd8301809fbb00aaff7dd3556605c3846de95c6444eaeed5ba77de WatchSource:0}: Error finding container 23df21d8c7bd8301809fbb00aaff7dd3556605c3846de95c6444eaeed5ba77de: Status 404 returned error can't find the container with id 23df21d8c7bd8301809fbb00aaff7dd3556605c3846de95c6444eaeed5ba77de Oct 30 18:30:01.035958 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.035907 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z"] Oct 30 18:30:01.066822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.066798 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr"] Oct 30 18:30:01.071242 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:30:01.071219 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676dcb93_cf50_4156_b4d9_998c3a90cd1a.slice/crio-2c07d59956a9c166150c8fb84157a9d569b53807fffed5bc99a5bbd3772ff454 WatchSource:0}: Error finding container 2c07d59956a9c166150c8fb84157a9d569b53807fffed5bc99a5bbd3772ff454: Status 404 returned error can't find the container with id 2c07d59956a9c166150c8fb84157a9d569b53807fffed5bc99a5bbd3772ff454 Oct 30 18:30:01.915382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.915347 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" event={"ID":"4febc95e-db96-4150-824e-c0838cce1ec7","Type":"ContainerStarted","Data":"550f5612ab0d9091493f0b0d6424e7fa1076908ec93381dcefcc671a4f3d7aba"} Oct 30 18:30:01.915980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.915950 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" event={"ID":"4febc95e-db96-4150-824e-c0838cce1ec7","Type":"ContainerStarted","Data":"0b074eb8095924bf30e082f30d1c6317de7cb9e109c7e0e0276f4b296da2a692"} Oct 30 18:30:01.917465 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.917440 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" event={"ID":"6cd8c46e-a5f0-438d-b75c-4fdc453b782f","Type":"ContainerStarted","Data":"fddea951b69453b31c14404781644369be4fe705a1650e79d659008d5bd6fa3b"} Oct 30 18:30:01.917616 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.917482 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" event={"ID":"6cd8c46e-a5f0-438d-b75c-4fdc453b782f","Type":"ContainerStarted","Data":"1778ed971ff7bc613e41564a12ecba8be91dff88b4ca88ca67ab20aadb9e0117"} Oct 30 18:30:01.919812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.919783 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" event={"ID":"676dcb93-cf50-4156-b4d9-998c3a90cd1a","Type":"ContainerStarted","Data":"3f4fe2e0afed8de220882216168ac5335b7c1e9e28007b2fef1462c67ca15a11"} Oct 30 18:30:01.919990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.919844 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" event={"ID":"676dcb93-cf50-4156-b4d9-998c3a90cd1a","Type":"ContainerStarted","Data":"2c07d59956a9c166150c8fb84157a9d569b53807fffed5bc99a5bbd3772ff454"} Oct 30 18:30:01.922336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.922310 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" event={"ID":"c16b7126-6234-4edf-bcf9-ee14e001720f","Type":"ContainerStarted","Data":"60e6c00be3aa6adf9cffc8df54c50ecf839c678c02450b10307713a5c5c18a60"} Oct 30 18:30:01.922430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.922356 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" event={"ID":"c16b7126-6234-4edf-bcf9-ee14e001720f","Type":"ContainerStarted","Data":"23df21d8c7bd8301809fbb00aaff7dd3556605c3846de95c6444eaeed5ba77de"} Oct 30 18:30:01.959191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.959072 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" podStartSLOduration=1.959054289 podStartE2EDuration="1.959054289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:01.956058592 +0000 UTC m=+1696.344586728" watchObservedRunningTime="2026-10-30 18:30:01.959054289 +0000 UTC m=+1696.347582445" Oct 30 18:30:01.998035 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:01.997965 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" podStartSLOduration=1.997929789 podStartE2EDuration="1.997929789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:01.995297344 +0000 UTC m=+1696.383825462" watchObservedRunningTime="2026-10-30 18:30:01.997929789 +0000 UTC m=+1696.386457907" Oct 30 18:30:02.036255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.036200 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" podStartSLOduration=2.036180825 podStartE2EDuration="2.036180825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:02.035116141 +0000 UTC m=+1696.423644259" watchObservedRunningTime="2026-10-30 18:30:02.036180825 +0000 UTC m=+1696.424708942" Oct 30 18:30:02.081163 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.081107 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:02.083492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.083446 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" podStartSLOduration=2.083433878 podStartE2EDuration="2.083433878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:02.08131607 +0000 UTC m=+1696.469844184" watchObservedRunningTime="2026-10-30 18:30:02.083433878 +0000 UTC m=+1696.471961994" Oct 30 18:30:02.554523 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.554497 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:02.651875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.651847 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:02.756107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.756004 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:02.931307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.931268 5624 generic.go:356] "Generic (PLEG): container finished" podID="4febc95e-db96-4150-824e-c0838cce1ec7" containerID="550f5612ab0d9091493f0b0d6424e7fa1076908ec93381dcefcc671a4f3d7aba" exitCode=60 Oct 30 18:30:02.931750 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.931354 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" event={"ID":"4febc95e-db96-4150-824e-c0838cce1ec7","Type":"ContainerDied","Data":"550f5612ab0d9091493f0b0d6424e7fa1076908ec93381dcefcc671a4f3d7aba"} Oct 30 18:30:02.931750 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.931629 5624 scope.go:122] "RemoveContainer" containerID="550f5612ab0d9091493f0b0d6424e7fa1076908ec93381dcefcc671a4f3d7aba" Oct 30 18:30:02.933893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.933870 5624 generic.go:356] "Generic (PLEG): container finished" podID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerID="fddea951b69453b31c14404781644369be4fe705a1650e79d659008d5bd6fa3b" exitCode=60 Oct 30 18:30:02.934077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.933996 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" event={"ID":"6cd8c46e-a5f0-438d-b75c-4fdc453b782f","Type":"ContainerDied","Data":"fddea951b69453b31c14404781644369be4fe705a1650e79d659008d5bd6fa3b"} Oct 30 18:30:02.934300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.934286 5624 scope.go:122] "RemoveContainer" containerID="fddea951b69453b31c14404781644369be4fe705a1650e79d659008d5bd6fa3b" Oct 30 18:30:02.935977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.935861 5624 generic.go:356] "Generic (PLEG): container finished" podID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerID="3f4fe2e0afed8de220882216168ac5335b7c1e9e28007b2fef1462c67ca15a11" exitCode=60 Oct 30 18:30:02.935977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.935909 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" event={"ID":"676dcb93-cf50-4156-b4d9-998c3a90cd1a","Type":"ContainerDied","Data":"3f4fe2e0afed8de220882216168ac5335b7c1e9e28007b2fef1462c67ca15a11"} Oct 30 18:30:02.936238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.936221 5624 scope.go:122] "RemoveContainer" containerID="3f4fe2e0afed8de220882216168ac5335b7c1e9e28007b2fef1462c67ca15a11" Oct 30 18:30:02.938156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.938126 5624 generic.go:356] "Generic (PLEG): container finished" podID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerID="b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f" exitCode=0 Oct 30 18:30:02.938288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.938212 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" event={"ID":"c2a49bec-3cdd-4178-be33-f8a0194365d0","Type":"ContainerDied","Data":"b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f"} Oct 30 18:30:02.938610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.938589 5624 scope.go:122] "RemoveContainer" containerID="b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f" Oct 30 18:30:02.945374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.945350 5624 generic.go:356] "Generic (PLEG): container finished" podID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerID="60e6c00be3aa6adf9cffc8df54c50ecf839c678c02450b10307713a5c5c18a60" exitCode=60 Oct 30 18:30:02.945500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.945441 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" event={"ID":"c16b7126-6234-4edf-bcf9-ee14e001720f","Type":"ContainerDied","Data":"60e6c00be3aa6adf9cffc8df54c50ecf839c678c02450b10307713a5c5c18a60"} Oct 30 18:30:02.945680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.945652 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:30:02.945762 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.945708 5624 scope.go:122] "RemoveContainer" containerID="60e6c00be3aa6adf9cffc8df54c50ecf839c678c02450b10307713a5c5c18a60" Oct 30 18:30:02.945762 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.945741 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:30:02.945883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:02.945797 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:30:03.663554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.663496 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:03.839366 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.839326 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:03.956758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.956721 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" event={"ID":"c2a49bec-3cdd-4178-be33-f8a0194365d0","Type":"ContainerStarted","Data":"9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac"} Oct 30 18:30:03.958917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.958883 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" event={"ID":"c16b7126-6234-4edf-bcf9-ee14e001720f","Type":"ContainerStarted","Data":"971e97260285bbec9dd7db1e718fa4d3a0fc0fb17247e3f36ccb9651d9c1d3a8"} Oct 30 18:30:03.961442 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.961413 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" event={"ID":"4febc95e-db96-4150-824e-c0838cce1ec7","Type":"ContainerStarted","Data":"0880d50e75899e78264d0581782de7cf2969045e72dafcfeb2c83f0cbc466cac"} Oct 30 18:30:03.963653 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.963617 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" event={"ID":"6cd8c46e-a5f0-438d-b75c-4fdc453b782f","Type":"ContainerStarted","Data":"24021b74468ee17fd114367d80c6e1ce3d549c33cb12dfaa4b8496f981459d38"} Oct 30 18:30:03.988959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.967837 5624 generic.go:356] "Generic (PLEG): container finished" podID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerID="d2bf83e861bd41a46919ae13b180cdf735333bef09510ee5f1aea9ac10119c11" exitCode=60 Oct 30 18:30:03.988959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.967883 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" event={"ID":"676dcb93-cf50-4156-b4d9-998c3a90cd1a","Type":"ContainerDied","Data":"d2bf83e861bd41a46919ae13b180cdf735333bef09510ee5f1aea9ac10119c11"} Oct 30 18:30:03.988959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.967954 5624 scope.go:122] "RemoveContainer" containerID="3f4fe2e0afed8de220882216168ac5335b7c1e9e28007b2fef1462c67ca15a11" Oct 30 18:30:03.988959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:03.968413 5624 scope.go:122] "RemoveContainer" containerID="d2bf83e861bd41a46919ae13b180cdf735333bef09510ee5f1aea9ac10119c11" Oct 30 18:30:03.988959 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:03.968783 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr_odh-ai-gateway-infra(676dcb93-cf50-4156-b4d9-998c3a90cd1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" Oct 30 18:30:04.289681 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.289648 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:30:04.293954 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.290027 5624 patch_prober.go:29] interesting pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:30:04.293954 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.290082 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 18:30:04.426072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.426032 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:30:04.556171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.556085 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:04.977743 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.977710 5624 generic.go:356] "Generic (PLEG): container finished" podID="4febc95e-db96-4150-824e-c0838cce1ec7" containerID="0880d50e75899e78264d0581782de7cf2969045e72dafcfeb2c83f0cbc466cac" exitCode=60 Oct 30 18:30:04.978252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.977796 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" event={"ID":"4febc95e-db96-4150-824e-c0838cce1ec7","Type":"ContainerDied","Data":"0880d50e75899e78264d0581782de7cf2969045e72dafcfeb2c83f0cbc466cac"} Oct 30 18:30:04.978252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.977868 5624 scope.go:122] "RemoveContainer" containerID="550f5612ab0d9091493f0b0d6424e7fa1076908ec93381dcefcc671a4f3d7aba" Oct 30 18:30:04.978252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.978104 5624 scope.go:122] "RemoveContainer" containerID="0880d50e75899e78264d0581782de7cf2969045e72dafcfeb2c83f0cbc466cac" Oct 30 18:30:04.978447 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:04.978422 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4_odh-ai-gateway-infra(4febc95e-db96-4150-824e-c0838cce1ec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" Oct 30 18:30:04.979946 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.979904 5624 generic.go:356] "Generic (PLEG): container finished" podID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerID="24021b74468ee17fd114367d80c6e1ce3d549c33cb12dfaa4b8496f981459d38" exitCode=60 Oct 30 18:30:04.980049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.980035 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" event={"ID":"6cd8c46e-a5f0-438d-b75c-4fdc453b782f","Type":"ContainerDied","Data":"24021b74468ee17fd114367d80c6e1ce3d549c33cb12dfaa4b8496f981459d38"} Oct 30 18:30:04.980358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.980295 5624 scope.go:122] "RemoveContainer" containerID="24021b74468ee17fd114367d80c6e1ce3d549c33cb12dfaa4b8496f981459d38" Oct 30 18:30:04.980590 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:04.980569 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-9cqqt_odh-ai-gateway-infra(6cd8c46e-a5f0-438d-b75c-4fdc453b782f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" Oct 30 18:30:04.982187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.982165 5624 scope.go:122] "RemoveContainer" containerID="d2bf83e861bd41a46919ae13b180cdf735333bef09510ee5f1aea9ac10119c11" Oct 30 18:30:04.982460 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:04.982435 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr_odh-ai-gateway-infra(676dcb93-cf50-4156-b4d9-998c3a90cd1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" Oct 30 18:30:04.983727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.983707 5624 generic.go:356] "Generic (PLEG): container finished" podID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerID="971e97260285bbec9dd7db1e718fa4d3a0fc0fb17247e3f36ccb9651d9c1d3a8" exitCode=60 Oct 30 18:30:04.983849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.983790 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" event={"ID":"c16b7126-6234-4edf-bcf9-ee14e001720f","Type":"ContainerDied","Data":"971e97260285bbec9dd7db1e718fa4d3a0fc0fb17247e3f36ccb9651d9c1d3a8"} Oct 30 18:30:04.995954 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:04.984007 5624 scope.go:122] "RemoveContainer" containerID="971e97260285bbec9dd7db1e718fa4d3a0fc0fb17247e3f36ccb9651d9c1d3a8" Oct 30 18:30:04.995954 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:04.984320 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z_odh-ai-gateway-infra(c16b7126-6234-4edf-bcf9-ee14e001720f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" Oct 30 18:30:05.033460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:05.033361 5624 scope.go:122] "RemoveContainer" containerID="fddea951b69453b31c14404781644369be4fe705a1650e79d659008d5bd6fa3b" Oct 30 18:30:05.055296 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:05.055228 5624 scope.go:122] "RemoveContainer" containerID="60e6c00be3aa6adf9cffc8df54c50ecf839c678c02450b10307713a5c5c18a60" Oct 30 18:30:05.288452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:05.288361 5624 patch_prober.go:29] interesting pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:30:05.288706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:05.288677 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 18:30:05.821879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:05.821844 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:30:06.024294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:06.024215 5624 scope.go:122] "RemoveContainer" containerID="0880d50e75899e78264d0581782de7cf2969045e72dafcfeb2c83f0cbc466cac" Oct 30 18:30:06.024829 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:06.024538 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4_odh-ai-gateway-infra(4febc95e-db96-4150-824e-c0838cce1ec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" Oct 30 18:30:06.027866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:06.027794 5624 scope.go:122] "RemoveContainer" containerID="24021b74468ee17fd114367d80c6e1ce3d549c33cb12dfaa4b8496f981459d38" Oct 30 18:30:06.028174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:06.028157 5624 scope.go:122] "RemoveContainer" containerID="971e97260285bbec9dd7db1e718fa4d3a0fc0fb17247e3f36ccb9651d9c1d3a8" Oct 30 18:30:06.028409 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:06.028391 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z_odh-ai-gateway-infra(c16b7126-6234-4edf-bcf9-ee14e001720f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" Oct 30 18:30:06.028555 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:06.028533 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-9cqqt_odh-ai-gateway-infra(6cd8c46e-a5f0-438d-b75c-4fdc453b782f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" Oct 30 18:30:06.241533 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:06.241497 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:06.294431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:06.294399 5624 patch_prober.go:29] interesting pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:30:06.294683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:06.294661 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 18:30:07.288337 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:07.288302 5624 patch_prober.go:29] interesting pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:30:07.288906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:07.288877 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 18:30:07.627783 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:07.627746 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:07.730398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:07.730362 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:07.849339 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:07.849302 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:08.288358 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:08.288315 5624 patch_prober.go:29] interesting pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:30:08.289005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:08.288967 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 18:30:08.461495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:08.461461 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:09.291555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:09.291524 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:30:09.292715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:09.292692 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:30:09.448509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:09.448481 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:10.365959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:10.365298 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:30:10.383304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:10.383262 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:10.488715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:10.488679 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:10.757226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:10.757179 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:10.767511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:10.767479 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z"] Oct 30 18:30:10.767717 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:10.767692 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" containerID="cri-o://578e4a7f2a7536419ccbdf6bc3ba621df02e50814c12580e545de90929dbd634" gracePeriod=30 Oct 30 18:30:11.239096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:11.239049 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:11.845575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:11.845538 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:30:11.995647 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:11.995616 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d866fec-8541-40d6-ae37-11dcfef32e46-kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/9d866fec-8541-40d6-ae37-11dcfef32e46-kube-api-access-r77qb\") pod \"9d866fec-8541-40d6-ae37-11dcfef32e46\" (UID: \"9d866fec-8541-40d6-ae37-11dcfef32e46\") " Oct 30 18:30:11.995853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:11.995791 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d866fec-8541-40d6-ae37-11dcfef32e46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d866fec-8541-40d6-ae37-11dcfef32e46-plugins-config-volume\") pod \"9d866fec-8541-40d6-ae37-11dcfef32e46\" (UID: \"9d866fec-8541-40d6-ae37-11dcfef32e46\") " Oct 30 18:30:11.996272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:11.996241 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d866fec-8541-40d6-ae37-11dcfef32e46-plugins-config-volume" pod "9d866fec-8541-40d6-ae37-11dcfef32e46" (UID: "9d866fec-8541-40d6-ae37-11dcfef32e46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:11.996447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:11.996429 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d866fec-8541-40d6-ae37-11dcfef32e46-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:11.998403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:11.998355 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d866fec-8541-40d6-ae37-11dcfef32e46-kube-api-access-r77qb" pod "9d866fec-8541-40d6-ae37-11dcfef32e46" (UID: "9d866fec-8541-40d6-ae37-11dcfef32e46"). InnerVolumeSpecName "kube-api-access-r77qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:12.098213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.098137 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77qb\" (UniqueName: \"kubernetes.io/projected/9d866fec-8541-40d6-ae37-11dcfef32e46-kube-api-access-r77qb\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:12.117859 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.117828 5624 generic.go:356] "Generic (PLEG): container finished" podID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerID="fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1" exitCode=0 Oct 30 18:30:12.118133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.117893 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" Oct 30 18:30:12.118133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.117925 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" event={"ID":"9d866fec-8541-40d6-ae37-11dcfef32e46","Type":"ContainerDied","Data":"fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1"} Oct 30 18:30:12.118133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.118012 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt" event={"ID":"9d866fec-8541-40d6-ae37-11dcfef32e46","Type":"ContainerDied","Data":"c6bab1b085255d69b7be81c031236b631e7b62e14738dc6d5b1e5bfe43868070"} Oct 30 18:30:12.118133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.118077 5624 scope.go:122] "RemoveContainer" containerID="fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1" Oct 30 18:30:12.133891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.133871 5624 scope.go:122] "RemoveContainer" containerID="fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1" Oct 30 18:30:12.134261 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:12.134236 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1\": container with ID starting with fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1 not found: ID does not exist" containerID="fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1" Oct 30 18:30:12.134341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.134272 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1"} err="failed to get container status \"fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1\": rpc error: code = NotFound desc = could not find container \"fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1\": container with ID starting with fc41f919c2838ac12122d1397788d26be2fb7597db742829b23d74420d4d81f1 not found: ID does not exist" Oct 30 18:30:12.174561 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.174529 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt"] Oct 30 18:30:12.188500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.188472 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8b6cf1-78687ccd67-xqmdt"] Oct 30 18:30:12.890237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:12.890209 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:13.793096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:13.793059 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" path="/var/lib/kubelet/pods/9d866fec-8541-40d6-ae37-11dcfef32e46/volumes" Oct 30 18:30:14.557773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:14.557730 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:14.783800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:14.783774 5624 scope.go:122] "RemoveContainer" containerID="971e97260285bbec9dd7db1e718fa4d3a0fc0fb17247e3f36ccb9651d9c1d3a8" Oct 30 18:30:14.784024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:14.783897 5624 scope.go:122] "RemoveContainer" containerID="0880d50e75899e78264d0581782de7cf2969045e72dafcfeb2c83f0cbc466cac" Oct 30 18:30:14.784334 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:14.784314 5624 scope.go:122] "RemoveContainer" containerID="d2bf83e861bd41a46919ae13b180cdf735333bef09510ee5f1aea9ac10119c11" Oct 30 18:30:15.141303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:15.141275 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" event={"ID":"676dcb93-cf50-4156-b4d9-998c3a90cd1a","Type":"ContainerStarted","Data":"d01077f079e3bf2ad3b0d819c756018965436af66ccce9e5610f62857e786c42"} Oct 30 18:30:15.790797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:15.790767 5624 scope.go:122] "RemoveContainer" containerID="24021b74468ee17fd114367d80c6e1ce3d549c33cb12dfaa4b8496f981459d38" Oct 30 18:30:15.876703 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:15.876671 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:16.157428 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.157394 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" event={"ID":"b5d96999-4d59-46f9-b7de-f4349a071b6c","Type":"ContainerDied","Data":"578e4a7f2a7536419ccbdf6bc3ba621df02e50814c12580e545de90929dbd634"} Oct 30 18:30:16.157598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.157348 5624 generic.go:356] "Generic (PLEG): container finished" podID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerID="578e4a7f2a7536419ccbdf6bc3ba621df02e50814c12580e545de90929dbd634" exitCode=0 Oct 30 18:30:16.157598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.157579 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" event={"ID":"b5d96999-4d59-46f9-b7de-f4349a071b6c","Type":"ContainerDied","Data":"9889e2490ea171bf9555ddb211066efced14ed8443e6d7c8a35b1dbed8f60dc6"} Oct 30 18:30:16.157735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.157614 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9889e2490ea171bf9555ddb211066efced14ed8443e6d7c8a35b1dbed8f60dc6" Oct 30 18:30:16.158053 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.158003 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:30:16.163677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.162349 5624 generic.go:356] "Generic (PLEG): container finished" podID="4febc95e-db96-4150-824e-c0838cce1ec7" containerID="638b5a054a17516fd80cd48188617d6373319d7d9f146bc6539ddd48eb370396" exitCode=60 Oct 30 18:30:16.163677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.162508 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" event={"ID":"4febc95e-db96-4150-824e-c0838cce1ec7","Type":"ContainerDied","Data":"638b5a054a17516fd80cd48188617d6373319d7d9f146bc6539ddd48eb370396"} Oct 30 18:30:16.163677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.162561 5624 scope.go:122] "RemoveContainer" containerID="0880d50e75899e78264d0581782de7cf2969045e72dafcfeb2c83f0cbc466cac" Oct 30 18:30:16.163677 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.163003 5624 scope.go:122] "RemoveContainer" containerID="638b5a054a17516fd80cd48188617d6373319d7d9f146bc6539ddd48eb370396" Oct 30 18:30:16.163677 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:16.163315 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4_odh-ai-gateway-infra(4febc95e-db96-4150-824e-c0838cce1ec7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" Oct 30 18:30:16.170381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.169278 5624 generic.go:356] "Generic (PLEG): container finished" podID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerID="d01077f079e3bf2ad3b0d819c756018965436af66ccce9e5610f62857e786c42" exitCode=60 Oct 30 18:30:16.170381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.169393 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" event={"ID":"676dcb93-cf50-4156-b4d9-998c3a90cd1a","Type":"ContainerDied","Data":"d01077f079e3bf2ad3b0d819c756018965436af66ccce9e5610f62857e786c42"} Oct 30 18:30:16.170381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.169732 5624 scope.go:122] "RemoveContainer" containerID="d01077f079e3bf2ad3b0d819c756018965436af66ccce9e5610f62857e786c42" Oct 30 18:30:16.170381 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:16.170056 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr_odh-ai-gateway-infra(676dcb93-cf50-4156-b4d9-998c3a90cd1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" Oct 30 18:30:16.173417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.173354 5624 generic.go:356] "Generic (PLEG): container finished" podID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerID="1794d2ec47d822e3161b8a1b6b3dfa844bc99640667b8eb29ce4acaba1049da2" exitCode=60 Oct 30 18:30:16.173511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.173432 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" event={"ID":"c16b7126-6234-4edf-bcf9-ee14e001720f","Type":"ContainerDied","Data":"1794d2ec47d822e3161b8a1b6b3dfa844bc99640667b8eb29ce4acaba1049da2"} Oct 30 18:30:16.173772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.173753 5624 scope.go:122] "RemoveContainer" containerID="1794d2ec47d822e3161b8a1b6b3dfa844bc99640667b8eb29ce4acaba1049da2" Oct 30 18:30:16.174057 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:16.174038 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z_odh-ai-gateway-infra(c16b7126-6234-4edf-bcf9-ee14e001720f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" Oct 30 18:30:16.182460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.182439 5624 scope.go:122] "RemoveContainer" containerID="d2bf83e861bd41a46919ae13b180cdf735333bef09510ee5f1aea9ac10119c11" Oct 30 18:30:16.255999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.253028 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr"] Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256313 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d96999-4d59-46f9-b7de-f4349a071b6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d96999-4d59-46f9-b7de-f4349a071b6c-istiod-ca-cert\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256387 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-token\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-token\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256451 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-credential-socket\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256491 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-certs\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256523 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-kube-api-access-w7w5x\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-kube-api-access-w7w5x\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256569 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-data\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256724 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-envoy\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256754 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-podinfo\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.271985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.256786 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-socket\") pod \"b5d96999-4d59-46f9-b7de-f4349a071b6c\" (UID: \"b5d96999-4d59-46f9-b7de-f4349a071b6c\") " Oct 30 18:30:16.288489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.288016 5624 scope.go:122] "RemoveContainer" containerID="971e97260285bbec9dd7db1e718fa4d3a0fc0fb17247e3f36ccb9651d9c1d3a8" Oct 30 18:30:16.289579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.289552 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d96999-4d59-46f9-b7de-f4349a071b6c-istiod-ca-cert" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:16.292021 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.291762 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-credential-socket" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.292363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.292293 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-data" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.292614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.292555 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-token" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.293985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.293950 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-certs" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.294131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.294116 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-socket" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.325418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.325381 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-kube-api-access-w7w5x" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "kube-api-access-w7w5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.330174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.330076 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-podinfo" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:16.331081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.331054 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-envoy" pod "b5d96999-4d59-46f9-b7de-f4349a071b6c" (UID: "b5d96999-4d59-46f9-b7de-f4349a071b6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.385144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385120 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5d96999-4d59-46f9-b7de-f4349a071b6c-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.385144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385144 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.385332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385157 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.385332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385170 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.385332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385183 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7w5x\" (UniqueName: \"kubernetes.io/projected/b5d96999-4d59-46f9-b7de-f4349a071b6c-kube-api-access-w7w5x\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.385332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385195 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.385332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385207 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.385332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385219 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5d96999-4d59-46f9-b7de-f4349a071b6c-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.385332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.385230 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5d96999-4d59-46f9-b7de-f4349a071b6c-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.453112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.453070 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:16.559113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.559036 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:16.664413 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:16.664365 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:17.117789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.117696 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:17.183511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.183479 5624 generic.go:356] "Generic (PLEG): container finished" podID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerID="911086a2fa17803de1adb2353a3c27c441738813c409df8d295c1496db420a38" exitCode=60 Oct 30 18:30:17.183704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.183549 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" event={"ID":"6cd8c46e-a5f0-438d-b75c-4fdc453b782f","Type":"ContainerDied","Data":"911086a2fa17803de1adb2353a3c27c441738813c409df8d295c1496db420a38"} Oct 30 18:30:17.183704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.183610 5624 scope.go:122] "RemoveContainer" containerID="24021b74468ee17fd114367d80c6e1ce3d549c33cb12dfaa4b8496f981459d38" Oct 30 18:30:17.184350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.184102 5624 scope.go:122] "RemoveContainer" containerID="911086a2fa17803de1adb2353a3c27c441738813c409df8d295c1496db420a38" Oct 30 18:30:17.184525 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:17.184474 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889750-9cqqt_odh-ai-gateway-infra(6cd8c46e-a5f0-438d-b75c-4fdc453b782f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" Oct 30 18:30:17.187390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.187372 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z" Oct 30 18:30:17.271764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.271733 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z"] Oct 30 18:30:17.291599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.291570 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-8j57z"] Oct 30 18:30:17.342460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.342428 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:17.370313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.370242 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z"] Oct 30 18:30:17.398124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.398082 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676dcb93-cf50-4156-b4d9-998c3a90cd1a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676dcb93-cf50-4156-b4d9-998c3a90cd1a-maas-api-service-ca\") pod \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\" (UID: \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\") " Oct 30 18:30:17.398780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.398650 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676dcb93-cf50-4156-b4d9-998c3a90cd1a-maas-api-service-ca" pod "676dcb93-cf50-4156-b4d9-998c3a90cd1a" (UID: "676dcb93-cf50-4156-b4d9-998c3a90cd1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:17.399254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.399235 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676dcb93-cf50-4156-b4d9-998c3a90cd1a-kube-api-access-d2bgg\" (UniqueName: \"kubernetes.io/projected/676dcb93-cf50-4156-b4d9-998c3a90cd1a-kube-api-access-d2bgg\") pod \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\" (UID: \"676dcb93-cf50-4156-b4d9-998c3a90cd1a\") " Oct 30 18:30:17.403872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.403846 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676dcb93-cf50-4156-b4d9-998c3a90cd1a-maas-api-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.406277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.406253 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676dcb93-cf50-4156-b4d9-998c3a90cd1a-kube-api-access-d2bgg" pod "676dcb93-cf50-4156-b4d9-998c3a90cd1a" (UID: "676dcb93-cf50-4156-b4d9-998c3a90cd1a"). InnerVolumeSpecName "kube-api-access-d2bgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:17.431067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.431036 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4"] Oct 30 18:30:17.509100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.509003 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2bgg\" (UniqueName: \"kubernetes.io/projected/676dcb93-cf50-4156-b4d9-998c3a90cd1a-kube-api-access-d2bgg\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.568497 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.568468 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:17.609876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.609852 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16b7126-6234-4edf-bcf9-ee14e001720f-kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/c16b7126-6234-4edf-bcf9-ee14e001720f-kube-api-access-bq78m\") pod \"c16b7126-6234-4edf-bcf9-ee14e001720f\" (UID: \"c16b7126-6234-4edf-bcf9-ee14e001720f\") " Oct 30 18:30:17.610007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.609966 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16b7126-6234-4edf-bcf9-ee14e001720f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16b7126-6234-4edf-bcf9-ee14e001720f-maas-api-service-ca\") pod \"c16b7126-6234-4edf-bcf9-ee14e001720f\" (UID: \"c16b7126-6234-4edf-bcf9-ee14e001720f\") " Oct 30 18:30:17.610537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.610507 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16b7126-6234-4edf-bcf9-ee14e001720f-maas-api-service-ca" pod "c16b7126-6234-4edf-bcf9-ee14e001720f" (UID: "c16b7126-6234-4edf-bcf9-ee14e001720f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:17.618214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.618173 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16b7126-6234-4edf-bcf9-ee14e001720f-kube-api-access-bq78m" pod "c16b7126-6234-4edf-bcf9-ee14e001720f" (UID: "c16b7126-6234-4edf-bcf9-ee14e001720f"). InnerVolumeSpecName "kube-api-access-bq78m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:17.631522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.631491 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:17.711217 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.711187 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4febc95e-db96-4150-824e-c0838cce1ec7-kube-api-access-2cgmr\" (UniqueName: \"kubernetes.io/projected/4febc95e-db96-4150-824e-c0838cce1ec7-kube-api-access-2cgmr\") pod \"4febc95e-db96-4150-824e-c0838cce1ec7\" (UID: \"4febc95e-db96-4150-824e-c0838cce1ec7\") " Oct 30 18:30:17.711528 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.711506 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4febc95e-db96-4150-824e-c0838cce1ec7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febc95e-db96-4150-824e-c0838cce1ec7-maas-api-service-ca\") pod \"4febc95e-db96-4150-824e-c0838cce1ec7\" (UID: \"4febc95e-db96-4150-824e-c0838cce1ec7\") " Oct 30 18:30:17.712392 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.712353 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4febc95e-db96-4150-824e-c0838cce1ec7-maas-api-service-ca" pod "4febc95e-db96-4150-824e-c0838cce1ec7" (UID: "4febc95e-db96-4150-824e-c0838cce1ec7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:17.715133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.715109 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq78m\" (UniqueName: \"kubernetes.io/projected/c16b7126-6234-4edf-bcf9-ee14e001720f-kube-api-access-bq78m\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.715232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.715136 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16b7126-6234-4edf-bcf9-ee14e001720f-maas-api-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.715232 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.715151 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febc95e-db96-4150-824e-c0838cce1ec7-maas-api-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.715315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.715299 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4febc95e-db96-4150-824e-c0838cce1ec7-kube-api-access-2cgmr" pod "4febc95e-db96-4150-824e-c0838cce1ec7" (UID: "4febc95e-db96-4150-824e-c0838cce1ec7"). InnerVolumeSpecName "kube-api-access-2cgmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:17.793895 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.793857 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" path="/var/lib/kubelet/pods/b5d96999-4d59-46f9-b7de-f4349a071b6c/volumes" Oct 30 18:30:17.816723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:17.816701 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cgmr\" (UniqueName: \"kubernetes.io/projected/4febc95e-db96-4150-824e-c0838cce1ec7-kube-api-access-2cgmr\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.196503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.196469 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" Oct 30 18:30:18.197022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.196502 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4" event={"ID":"4febc95e-db96-4150-824e-c0838cce1ec7","Type":"ContainerDied","Data":"0b074eb8095924bf30e082f30d1c6317de7cb9e109c7e0e0276f4b296da2a692"} Oct 30 18:30:18.197022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.196571 5624 scope.go:122] "RemoveContainer" containerID="638b5a054a17516fd80cd48188617d6373319d7d9f146bc6539ddd48eb370396" Oct 30 18:30:18.201321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.201295 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" event={"ID":"676dcb93-cf50-4156-b4d9-998c3a90cd1a","Type":"ContainerDied","Data":"2c07d59956a9c166150c8fb84157a9d569b53807fffed5bc99a5bbd3772ff454"} Oct 30 18:30:18.201452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.201305 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr" Oct 30 18:30:18.202888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.202866 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" event={"ID":"c16b7126-6234-4edf-bcf9-ee14e001720f","Type":"ContainerDied","Data":"23df21d8c7bd8301809fbb00aaff7dd3556605c3846de95c6444eaeed5ba77de"} Oct 30 18:30:18.203005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.202971 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z" Oct 30 18:30:18.208181 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.208164 5624 scope.go:122] "RemoveContainer" containerID="d01077f079e3bf2ad3b0d819c756018965436af66ccce9e5610f62857e786c42" Oct 30 18:30:18.218314 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.218294 5624 scope.go:122] "RemoveContainer" containerID="1794d2ec47d822e3161b8a1b6b3dfa844bc99640667b8eb29ce4acaba1049da2" Oct 30 18:30:18.236861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.236840 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt"] Oct 30 18:30:18.259003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.258976 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4"] Oct 30 18:30:18.267675 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.267647 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0f0c50-29889750-zsbd4"] Oct 30 18:30:18.296998 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.295174 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr"] Oct 30 18:30:18.298277 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.298249 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c22995-29889750-9fpkr"] Oct 30 18:30:18.335733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.335694 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z"] Oct 30 18:30:18.343406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.343375 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-69c44c-29889750-6c44z"] Oct 30 18:30:18.410929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.410907 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:18.525312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.525219 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-kube-api-access-gx4jz\" (UniqueName: \"kubernetes.io/projected/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-kube-api-access-gx4jz\") pod \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\" (UID: \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\") " Oct 30 18:30:18.525312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.525302 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-maas-api-service-ca\") pod \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\" (UID: \"6cd8c46e-a5f0-438d-b75c-4fdc453b782f\") " Oct 30 18:30:18.526112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.526083 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-maas-api-service-ca" pod "6cd8c46e-a5f0-438d-b75c-4fdc453b782f" (UID: "6cd8c46e-a5f0-438d-b75c-4fdc453b782f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:18.527745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.527722 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-kube-api-access-gx4jz" pod "6cd8c46e-a5f0-438d-b75c-4fdc453b782f" (UID: "6cd8c46e-a5f0-438d-b75c-4fdc453b782f"). InnerVolumeSpecName "kube-api-access-gx4jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:18.536362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.536324 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:18.627804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.626779 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-maas-api-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.627804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:18.626815 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx4jz\" (UniqueName: \"kubernetes.io/projected/6cd8c46e-a5f0-438d-b75c-4fdc453b782f-kube-api-access-gx4jz\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:19.218370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.218341 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" Oct 30 18:30:19.218370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.218351 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt" event={"ID":"6cd8c46e-a5f0-438d-b75c-4fdc453b782f","Type":"ContainerDied","Data":"1778ed971ff7bc613e41564a12ecba8be91dff88b4ca88ca67ab20aadb9e0117"} Oct 30 18:30:19.218920 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.218420 5624 scope.go:122] "RemoveContainer" containerID="911086a2fa17803de1adb2353a3c27c441738813c409df8d295c1496db420a38" Oct 30 18:30:19.274038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.274014 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt"] Oct 30 18:30:19.286999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.286973 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889750-9cqqt"] Oct 30 18:30:19.383270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.383230 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:19.486777 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.485191 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:19.793395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.793321 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" path="/var/lib/kubelet/pods/4febc95e-db96-4150-824e-c0838cce1ec7/volumes" Oct 30 18:30:19.793967 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.793947 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" path="/var/lib/kubelet/pods/676dcb93-cf50-4156-b4d9-998c3a90cd1a/volumes" Oct 30 18:30:19.794492 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.794475 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" path="/var/lib/kubelet/pods/6cd8c46e-a5f0-438d-b75c-4fdc453b782f/volumes" Oct 30 18:30:19.795428 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:19.795409 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" path="/var/lib/kubelet/pods/c16b7126-6234-4edf-bcf9-ee14e001720f/volumes" Oct 30 18:30:19.918256 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:19.918218 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:30:20.161687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:20.161658 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:20.446226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:20.446124 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:21.092025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:21.091983 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:21.191822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:21.191788 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:21.879488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:21.879447 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:21.980710 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:21.980675 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:23.553901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:23.553825 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:24.302643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:24.302612 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:24.410338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:24.410308 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:25.074996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:25.074912 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:25.246066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:25.246031 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:25.542010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:25.541978 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:25.645097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:25.645064 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:25.726068 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:25.726040 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw"] Oct 30 18:30:25.726549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:25.726515 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" containerID="cri-o://f50c670664203c625a5333421eeaf48fffec42081705a541a634e9a70d9fe6e9" gracePeriod=30 Oct 30 18:30:25.751435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:25.751406 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:26.137877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.137843 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj"] Oct 30 18:30:26.138425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138410 5624 cpu_manager.go:414] "containerMap: removing container" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:30:26.138475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138425 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:30:26.138475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138443 5624 cpu_manager.go:414] "containerMap: removing container" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:30:26.138475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138448 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:30:26.138475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138456 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:30:26.138475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138462 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:30:26.138475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138471 5624 cpu_manager.go:414] "containerMap: removing container" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" Oct 30 18:30:26.138475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138476 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138482 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138487 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138496 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138500 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138511 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138516 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138526 5624 cpu_manager.go:414] "containerMap: removing container" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138530 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138537 5624 cpu_manager.go:414] "containerMap: removing container" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138541 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138553 5624 cpu_manager.go:414] "containerMap: removing container" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138558 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138570 5624 cpu_manager.go:414] "containerMap: removing container" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138575 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138653 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138661 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138669 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138675 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138680 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d96999-4d59-46f9-b7de-f4349a071b6c" containerName="istio-proxy" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138684 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138690 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138697 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.138693 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138703 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:30:26.139450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138709 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="4febc95e-db96-4150-824e-c0838cce1ec7" containerName="cleanup" Oct 30 18:30:26.139450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138715 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:30:26.139450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138721 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d866fec-8541-40d6-ae37-11dcfef32e46" containerName="payload-pre-processing" Oct 30 18:30:26.139450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138727 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:30:26.139450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.138733 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:30:26.144174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.144154 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.165070 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.165043 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj"] Oct 30 18:30:26.172690 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.172670 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1b0848-openshift-default-dockercfg-cw6rr\"" Oct 30 18:30:26.240538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.240497 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-data\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.240721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.240616 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-token\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.240721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.240650 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.240721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.240669 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-certs\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.240905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.240872 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pdx8\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-kube-api-access-8pdx8\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.240997 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.240971 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istiod-ca-cert\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.241066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.241015 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-podinfo\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.241129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.241090 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-credential-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.241187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.241135 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-envoy\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342112 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-token\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342150 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342166 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-certs\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342245 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pdx8\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-kube-api-access-8pdx8\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342291 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istiod-ca-cert\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342324 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-podinfo\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342371 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-credential-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342412 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-envoy\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342472 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-data\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342591 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342748 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-credential-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.342851 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342801 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-data\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.343051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.342977 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-certs\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.343128 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.343106 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istiod-ca-cert\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.344716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.344697 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-envoy\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.345052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.345028 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-podinfo\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.364365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.364343 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-token\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.364607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.364583 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pdx8\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-kube-api-access-8pdx8\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.458127 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.458041 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:26.780472 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:30:26.780426 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11579cbd_3b9f_4f8f_865f_2d46cb24901d.slice/crio-a2445192326b0acbdd917cb5f45e50ceb76d25ddb2f6e88da918dd89e4584db0 WatchSource:0}: Error finding container a2445192326b0acbdd917cb5f45e50ceb76d25ddb2f6e88da918dd89e4584db0: Status 404 returned error can't find the container with id a2445192326b0acbdd917cb5f45e50ceb76d25ddb2f6e88da918dd89e4584db0 Oct 30 18:30:26.782084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.782060 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj"] Oct 30 18:30:26.782688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.782660 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:30:26.782742 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.782727 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:30:26.782778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.782754 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:30:26.789415 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:26.789392 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:27.028604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:27.028521 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:27.288635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:27.288534 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" event={"ID":"11579cbd-3b9f-4f8f-865f-2d46cb24901d","Type":"ContainerStarted","Data":"ca5f6f6cc3276c279017e32cf3f78bb471070b4fd581575f8b06aca6830780bd"} Oct 30 18:30:27.288635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:27.288603 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" event={"ID":"11579cbd-3b9f-4f8f-865f-2d46cb24901d","Type":"ContainerStarted","Data":"a2445192326b0acbdd917cb5f45e50ceb76d25ddb2f6e88da918dd89e4584db0"} Oct 30 18:30:27.335084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:27.335032 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" podStartSLOduration=1.335017753 podStartE2EDuration="1.335017753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:27.332167356 +0000 UTC m=+1721.720695475" watchObservedRunningTime="2026-10-30 18:30:27.335017753 +0000 UTC m=+1721.723545870" Oct 30 18:30:27.458610 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:27.458572 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:27.460467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:27.460439 5624 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:30:27.460631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:27.460492 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 18:30:28.459067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:28.459027 5624 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:30:28.459494 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:28.459085 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 18:30:28.721039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:28.720576 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:29.458685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:29.458650 5624 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:30:29.458869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:29.458700 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 18:30:30.342959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:30.342819 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:30.414956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:30.411218 5624 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:30:30.414956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:30.411276 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:30:30.458551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:30.458521 5624 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:30:30.458806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:30.458778 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 18:30:31.354111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.354082 5624 generic.go:356] "Generic (PLEG): container finished" podID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerID="f50c670664203c625a5333421eeaf48fffec42081705a541a634e9a70d9fe6e9" exitCode=0 Oct 30 18:30:31.354592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.354574 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" event={"ID":"56a9e445-860a-46c5-80bd-6ed297d09b77","Type":"ContainerDied","Data":"f50c670664203c625a5333421eeaf48fffec42081705a541a634e9a70d9fe6e9"} Oct 30 18:30:31.354717 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.354707 5624 scope.go:122] "RemoveContainer" containerID="54f17b7248f7563e7f291f5874d4a5b435a9dcfe1dc08af6b6723bc42f9c38b9" Oct 30 18:30:31.420179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.420144 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:30:31.459197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.459167 5624 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:30:31.459349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.459213 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 18:30:31.468873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.468842 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-token\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-token\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.468881 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-socket\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.468928 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-kube-api-access-hfctc\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-kube-api-access-hfctc\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.468993 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-data\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469013 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a9e445-860a-46c5-80bd-6ed297d09b77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a9e445-860a-46c5-80bd-6ed297d09b77-istiod-ca-cert\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469098 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-credential-socket\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469157 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-podinfo\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469272 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469250 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-certs\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469268 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-socket" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.469451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469378 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-data" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.469486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469444 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-envoy\") pod \"56a9e445-860a-46c5-80bd-6ed297d09b77\" (UID: \"56a9e445-860a-46c5-80bd-6ed297d09b77\") " Oct 30 18:30:31.469486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469443 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-credential-socket" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.469555 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.469483 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a9e445-860a-46c5-80bd-6ed297d09b77-istiod-ca-cert" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:31.470117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.470086 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.470117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.470114 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.470261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.470115 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-certs" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.470261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.470128 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56a9e445-860a-46c5-80bd-6ed297d09b77-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.470261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.470144 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.471869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.471844 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-token" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:31.475656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.475570 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-envoy" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.475656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.475630 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-podinfo" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:31.475872 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.475849 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-kube-api-access-hfctc" pod "56a9e445-860a-46c5-80bd-6ed297d09b77" (UID: "56a9e445-860a-46c5-80bd-6ed297d09b77"). InnerVolumeSpecName "kube-api-access-hfctc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:31.571012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.570986 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfctc\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-kube-api-access-hfctc\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.571012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.571011 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.571199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.571021 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.571199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.571029 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.571199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:31.571037 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a9e445-860a-46c5-80bd-6ed297d09b77-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:30:32.234741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:32.234709 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:32.364478 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:32.364447 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" Oct 30 18:30:32.364964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:32.364484 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw" event={"ID":"56a9e445-860a-46c5-80bd-6ed297d09b77","Type":"ContainerDied","Data":"cc360df749b357d3024640b7b8a8c7eb840ad9563468f82eec9c63de5058ce3f"} Oct 30 18:30:32.364964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:32.364548 5624 scope.go:122] "RemoveContainer" containerID="f50c670664203c625a5333421eeaf48fffec42081705a541a634e9a70d9fe6e9" Oct 30 18:30:32.445695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:32.445662 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw"] Oct 30 18:30:32.456866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:32.456830 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-9qhvw"] Oct 30 18:30:32.463858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:32.461307 5624 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 18:30:32.463858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:32.461367 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 18:30:33.462175 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:33.462148 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:33.462587 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:33.462561 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:33.463216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:33.463196 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:30:33.792521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:33.792445 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" path="/var/lib/kubelet/pods/56a9e445-860a-46c5-80bd-6ed297d09b77/volumes" Oct 30 18:30:33.933333 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:33.933298 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:35.527728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:35.527672 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:36.508676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:36.508641 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:37.128228 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:37.128200 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:38.498579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:38.498544 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:30:38.815248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:38.815168 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:40.443303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:40.443268 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:42.029716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:42.029681 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:42.578018 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:42.577982 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:43.692813 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:43.692778 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:45.236407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:45.236364 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:46.931064 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:46.931027 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:48.633161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:48.633133 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:49.914462 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:30:49.914429 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:30:50.242212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:50.242136 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:51.754045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:51.754006 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:53.336058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:53.336025 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:54.788800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:54.788754 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:30:54.885858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:54.885821 5624 scope.go:122] "RemoveContainer" containerID="578e4a7f2a7536419ccbdf6bc3ba621df02e50814c12580e545de90929dbd634" Oct 30 18:30:54.946095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:54.946068 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:56.625993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:56.625945 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:58.232238 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:58.232203 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:59.802751 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:30:59.802709 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:01.328755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:01.328719 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:02.917496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:02.917464 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:04.514774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:04.514738 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:06.040890 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:06.040855 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:07.645474 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:07.645443 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:09.342849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:09.342818 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:11.031641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:11.031608 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:12.602222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:12.602191 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:14.134448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:14.134399 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:15.638880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:15.638843 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:17.234328 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:17.234291 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:18.860411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:18.860358 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:19.645578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:19.645546 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:31:19.920732 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:31:19.920690 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:31:20.346952 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:20.346849 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:21.941226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:21.941191 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:23.629360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:23.629321 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:25.237860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:25.237825 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:26.826126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:26.826064 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:28.442783 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:28.442738 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:29.952962 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:29.952908 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:31.628397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:31.628363 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:32.180216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:32.180187 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:33.231157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:33.231109 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:34.450191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:34.450157 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:31:34.805735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:34.805636 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:36.327365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:36.327330 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:37.953462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:37.953402 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:39.592495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:39.592465 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:41.178874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:41.178809 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:42.027033 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:42.027000 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9"] Oct 30 18:31:42.027219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:42.027199 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" containerID="cri-o://24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129" gracePeriod=30 Oct 30 18:31:42.709180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:42.709146 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:42.809073 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:42.809041 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:44.346679 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:44.346646 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:45.950040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:45.950003 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:47.299522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.299460 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:31:47.417013 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.416976 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-envoy\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417083 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-socket\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417131 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-credential-socket\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417208 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417181 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-data\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417247 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-kube-api-access-kv7ps\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-kube-api-access-kv7ps\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417288 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-podinfo\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417416 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417344 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-certs\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417465 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-token\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-token\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417498 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istiod-ca-cert\") pod \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\" (UID: \"cd73c927-afa4-4023-bf6b-a04af2ea3a79\") " Oct 30 18:31:47.417716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417667 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-data" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:47.417980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.417929 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-socket" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:47.418084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.418011 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.418188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.418153 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-credential-socket" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:47.418395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.418375 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-certs" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:47.419219 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.419197 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istiod-ca-cert" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:47.420657 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.420619 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-kube-api-access-kv7ps" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "kube-api-access-kv7ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:47.422720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.422688 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-podinfo" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:31:47.422830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.422778 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-envoy" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:47.424631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.424583 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-token" pod "cd73c927-afa4-4023-bf6b-a04af2ea3a79" (UID: "cd73c927-afa4-4023-bf6b-a04af2ea3a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:47.519979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.519917 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7ps\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-kube-api-access-kv7ps\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.519979 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.519981 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.520244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.519999 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.520244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.520014 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.520244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.520028 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.520244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.520042 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.520244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.520056 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.520244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.520069 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd73c927-afa4-4023-bf6b-a04af2ea3a79-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:31:47.625812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.625773 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:47.943264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.943185 5624 generic.go:356] "Generic (PLEG): container finished" podID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerID="24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129" exitCode=0 Oct 30 18:31:47.943264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.943219 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" event={"ID":"cd73c927-afa4-4023-bf6b-a04af2ea3a79","Type":"ContainerDied","Data":"24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129"} Oct 30 18:31:47.943264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.943242 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" Oct 30 18:31:47.943581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.943275 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9" event={"ID":"cd73c927-afa4-4023-bf6b-a04af2ea3a79","Type":"ContainerDied","Data":"18680fb00f6cb7709256c14b4ab6f77ea90805a37859ea5510f80256b0a20099"} Oct 30 18:31:47.943581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.943300 5624 scope.go:122] "RemoveContainer" containerID="24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129" Oct 30 18:31:47.954320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.954300 5624 scope.go:122] "RemoveContainer" containerID="24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129" Oct 30 18:31:47.954610 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:31:47.954593 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129\": container with ID starting with 24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129 not found: ID does not exist" containerID="24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129" Oct 30 18:31:47.954685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.954618 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129"} err="failed to get container status \"24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129\": rpc error: code = NotFound desc = could not find container \"24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129\": container with ID starting with 24203626d02d230c01bd6b7d6c08faa08bcc95362cb7e2e0ba5ce2ab88a7b129 not found: ID does not exist" Oct 30 18:31:48.001981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:47.999792 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9"] Oct 30 18:31:48.003398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:48.003371 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-z4hb9"] Oct 30 18:31:49.328527 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:49.328486 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:49.802720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:49.802684 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" path="/var/lib/kubelet/pods/cd73c927-afa4-4023-bf6b-a04af2ea3a79/volumes" Oct 30 18:31:50.009157 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:31:50.009121 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:31:51.024520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:51.024480 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:51.302521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:51.302437 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:52.538369 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:52.538334 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:53.807707 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:53.807630 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:54.246226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:54.246191 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:55.845380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:55.845347 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:57.570834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:57.570788 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:59.225498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:31:59.225462 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:00.707704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:00.707667 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:01.392680 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:01.392645 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:32:02.216817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:02.216783 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:03.749172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:03.749139 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:05.325764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:05.325730 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:06.930370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:06.930324 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:07.025595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:07.025554 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:07.309370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:07.309290 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:08.843831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:08.843802 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:10.524210 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:10.524179 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:12.036970 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:12.036924 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:13.719449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:13.719415 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:15.345823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:15.345790 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:16.930390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:16.930355 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:18.182406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:18.182370 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:18.292566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:18.292518 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:18.447197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:18.447128 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:19.293983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:19.293924 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:19.917854 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:32:19.917810 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:32:20.147832 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:20.147802 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:21.577678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:21.577642 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:32:21.768820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:21.768771 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:23.426506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:23.426469 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:24.979406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:24.979371 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:26.252049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.252011 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh"] Oct 30 18:32:26.252543 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.252297 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" podUID="49fd0eb0-333a-431b-9395-1f2629097152" containerName="maas-api" containerID="cri-o://bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c" gracePeriod=30 Oct 30 18:32:26.309916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.309863 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:26.772868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.772841 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:32:26.812686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.812436 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls\") pod \"49fd0eb0-333a-431b-9395-1f2629097152\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " Oct 30 18:32:26.812686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.812479 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-metrics-certs\") pod \"49fd0eb0-333a-431b-9395-1f2629097152\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " Oct 30 18:32:26.812686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.812614 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49fd0eb0-333a-431b-9395-1f2629097152-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fd0eb0-333a-431b-9395-1f2629097152-openshift-service-ca\") pod \"49fd0eb0-333a-431b-9395-1f2629097152\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " Oct 30 18:32:26.812989 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.812735 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49fd0eb0-333a-431b-9395-1f2629097152-kube-api-access-srqdn\" (UniqueName: \"kubernetes.io/projected/49fd0eb0-333a-431b-9395-1f2629097152-kube-api-access-srqdn\") pod \"49fd0eb0-333a-431b-9395-1f2629097152\" (UID: \"49fd0eb0-333a-431b-9395-1f2629097152\") " Oct 30 18:32:26.813100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.813068 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49fd0eb0-333a-431b-9395-1f2629097152-openshift-service-ca" pod "49fd0eb0-333a-431b-9395-1f2629097152" (UID: "49fd0eb0-333a-431b-9395-1f2629097152"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:26.813341 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.813321 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49fd0eb0-333a-431b-9395-1f2629097152-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:26.813514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.813483 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:26.815333 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.815306 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls" pod "49fd0eb0-333a-431b-9395-1f2629097152" (UID: "49fd0eb0-333a-431b-9395-1f2629097152"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:26.815540 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.815520 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-metrics-certs" pod "49fd0eb0-333a-431b-9395-1f2629097152" (UID: "49fd0eb0-333a-431b-9395-1f2629097152"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:26.816244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.816219 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fd0eb0-333a-431b-9395-1f2629097152-kube-api-access-srqdn" pod "49fd0eb0-333a-431b-9395-1f2629097152" (UID: "49fd0eb0-333a-431b-9395-1f2629097152"). InnerVolumeSpecName "kube-api-access-srqdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:26.911188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.911154 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:26.914719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.914695 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:26.914719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.914715 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49fd0eb0-333a-431b-9395-1f2629097152-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:26.914719 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:26.914724 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srqdn\" (UniqueName: \"kubernetes.io/projected/49fd0eb0-333a-431b-9395-1f2629097152-kube-api-access-srqdn\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:27.013499 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.011866 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:27.342101 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.342061 5624 generic.go:356] "Generic (PLEG): container finished" podID="49fd0eb0-333a-431b-9395-1f2629097152" containerID="bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c" exitCode=0 Oct 30 18:32:27.342595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.342122 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" Oct 30 18:32:27.342595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.342173 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" event={"ID":"49fd0eb0-333a-431b-9395-1f2629097152","Type":"ContainerDied","Data":"bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c"} Oct 30 18:32:27.342595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.342223 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" event={"ID":"49fd0eb0-333a-431b-9395-1f2629097152","Type":"ContainerDied","Data":"16566956e1eb04972a6d1cec3c2168f6df9b087f39067b5c2dc78b7816163cd1"} Oct 30 18:32:27.342595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.342257 5624 scope.go:122] "RemoveContainer" containerID="bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c" Oct 30 18:32:27.353975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.353925 5624 scope.go:122] "RemoveContainer" containerID="bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c" Oct 30 18:32:27.354262 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:32:27.354242 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c\": container with ID starting with bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c not found: ID does not exist" containerID="bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c" Oct 30 18:32:27.354342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.354270 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c"} err="failed to get container status \"bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c\": rpc error: code = NotFound desc = could not find container \"bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c\": container with ID starting with bd4891f9dd5e2a3a8709d7db23238de4d7767ea39dd439e1a7ddf71a3c92393c not found: ID does not exist" Oct 30 18:32:27.398337 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.398308 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh"] Oct 30 18:32:27.411000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.410977 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh"] Oct 30 18:32:27.536391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.536362 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:27.793380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:27.793336 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49fd0eb0-333a-431b-9395-1f2629097152" path="/var/lib/kubelet/pods/49fd0eb0-333a-431b-9395-1f2629097152/volumes" Oct 30 18:32:28.531247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:28.531216 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:29.105215 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:29.105181 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:29.207918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:29.207879 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:29.698625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:29.698570 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0f0c50-9687f79c8-twjlh" podUID="49fd0eb0-333a-431b-9395-1f2629097152" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.141:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:32:29.866775 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:29.866744 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24"] Oct 30 18:32:29.867005 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:29.866978 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" podUID="7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" containerName="maas-api" containerID="cri-o://688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43" gracePeriod=30 Oct 30 18:32:29.921283 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:29.921242 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:30.036770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:30.036690 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r"] Oct 30 18:32:30.092430 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:30.092397 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:30.309670 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:30.309582 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:30.649702 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:30.649669 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:31.162156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.162046 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:32:31.167482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.167439 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-metrics-certs\") pod \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " Oct 30 18:32:31.167633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.167594 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-openshift-service-ca\") pod \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " Oct 30 18:32:31.167633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.167627 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-maas-api-tls\") pod \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " Oct 30 18:32:31.167769 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.167750 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-kube-api-access-xmbnx\" (UniqueName: \"kubernetes.io/projected/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-kube-api-access-xmbnx\") pod \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\" (UID: \"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80\") " Oct 30 18:32:31.169512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.169484 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-openshift-service-ca" pod "7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" (UID: "7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:31.179203 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.179162 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-kube-api-access-xmbnx" pod "7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" (UID: "7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80"). InnerVolumeSpecName "kube-api-access-xmbnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:31.195434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.195245 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-maas-api-tls" pod "7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" (UID: "7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:31.195434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.195349 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-metrics-certs" pod "7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" (UID: "7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:31.269287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.269253 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmbnx\" (UniqueName: \"kubernetes.io/projected/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-kube-api-access-xmbnx\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:31.269287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.269281 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:31.269287 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.269290 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:31.269520 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.269300 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:31.386151 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.386117 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:31.386371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.386178 5624 generic.go:356] "Generic (PLEG): container finished" podID="7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" containerID="688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43" exitCode=0 Oct 30 18:32:31.386371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.386233 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" Oct 30 18:32:31.386371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.386230 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" event={"ID":"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80","Type":"ContainerDied","Data":"688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43"} Oct 30 18:32:31.386371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.386361 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24" event={"ID":"7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80","Type":"ContainerDied","Data":"5575360b59b9d1cbe03ca8421b8af97c6b0e13748454dcc9e8a083cc4247f235"} Oct 30 18:32:31.386614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.386393 5624 scope.go:122] "RemoveContainer" containerID="688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43" Oct 30 18:32:31.397509 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.397489 5624 scope.go:122] "RemoveContainer" containerID="688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43" Oct 30 18:32:31.397818 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:32:31.397790 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43\": container with ID starting with 688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43 not found: ID does not exist" containerID="688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43" Oct 30 18:32:31.397950 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.397820 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43"} err="failed to get container status \"688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43\": rpc error: code = NotFound desc = could not find container \"688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43\": container with ID starting with 688fe8cff586b4a7367b91a0ec951e6af64ede45cfa912998778748e8bfdaf43 not found: ID does not exist" Oct 30 18:32:31.449122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.449026 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24"] Oct 30 18:32:31.450623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.450603 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0848-555c7688f9-5kk24"] Oct 30 18:32:31.475423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.475398 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:31.813006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:31.808405 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" path="/var/lib/kubelet/pods/7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80/volumes" Oct 30 18:32:32.004686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:32.004649 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:32.536298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:32.536264 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:33.902845 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:33.902809 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:34.015959 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.011793 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:34.278839 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.278766 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:34.391822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.391624 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:34.869488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.869451 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt"] Oct 30 18:32:34.870010 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.869997 5624 cpu_manager.go:414] "containerMap: removing container" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870011 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870024 5624 cpu_manager.go:414] "containerMap: removing container" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870030 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870049 5624 cpu_manager.go:414] "containerMap: removing container" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870054 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676dcb93-cf50-4156-b4d9-998c3a90cd1a" containerName="cleanup" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870068 5624 cpu_manager.go:414] "containerMap: removing container" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870072 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16b7126-6234-4edf-bcf9-ee14e001720f" containerName="cleanup" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870081 5624 cpu_manager.go:414] "containerMap: removing container" podUID="7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" containerName="maas-api" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870088 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" containerName="maas-api" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870098 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870103 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd8c46e-a5f0-438d-b75c-4fdc453b782f" containerName="cleanup" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870114 5624 cpu_manager.go:414] "containerMap: removing container" podUID="49fd0eb0-333a-431b-9395-1f2629097152" containerName="maas-api" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870118 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49fd0eb0-333a-431b-9395-1f2629097152" containerName="maas-api" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870132 5624 cpu_manager.go:414] "containerMap: removing container" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870136 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870211 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5f7d6b-4999-41bb-9bff-1e74f6fc2b80" containerName="maas-api" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870221 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="49fd0eb0-333a-431b-9395-1f2629097152" containerName="maas-api" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870229 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd73c927-afa4-4023-bf6b-a04af2ea3a79" containerName="istio-proxy" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870235 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" Oct 30 18:32:34.870316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.870241 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a9e445-860a-46c5-80bd-6ed297d09b77" containerName="istio-proxy" Oct 30 18:32:34.873198 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.873180 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:34.893171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.893144 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt"] Oct 30 18:32:34.901908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.901882 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:32:34.902047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.901976 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:32:34.924541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.924509 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6406ed84-abfd-457a-be38-7354061379c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:34.924883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.924678 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvgl\" (UniqueName: \"kubernetes.io/projected/6406ed84-abfd-457a-be38-7354061379c6-kube-api-access-dvvgl\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:34.924883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:34.924803 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6406ed84-abfd-457a-be38-7354061379c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:35.025927 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.025896 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6406ed84-abfd-457a-be38-7354061379c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:35.026101 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.026007 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6406ed84-abfd-457a-be38-7354061379c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:35.026101 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.026098 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvgl\" (UniqueName: \"kubernetes.io/projected/6406ed84-abfd-457a-be38-7354061379c6-kube-api-access-dvvgl\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:35.026510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.026487 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6406ed84-abfd-457a-be38-7354061379c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:35.028390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.028371 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6406ed84-abfd-457a-be38-7354061379c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:35.037313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.037279 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" podUID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" containerName="payload-pre-processing" containerID="cri-o://57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181" gracePeriod=25 Oct 30 18:32:35.056158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.056133 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" podUID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:32:35.075919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.075896 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvgl\" (UniqueName: \"kubernetes.io/projected/6406ed84-abfd-457a-be38-7354061379c6-kube-api-access-dvvgl\") pod \"maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:35.187251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.187220 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:35.494473 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:32:35.494433 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6406ed84_abfd_457a_be38_7354061379c6.slice/crio-1047ca39c9c82a25b8b75139897f7bb191c0b078df0738f86dd6d3f06d8823c7 WatchSource:0}: Error finding container 1047ca39c9c82a25b8b75139897f7bb191c0b078df0738f86dd6d3f06d8823c7: Status 404 returned error can't find the container with id 1047ca39c9c82a25b8b75139897f7bb191c0b078df0738f86dd6d3f06d8823c7 Oct 30 18:32:35.495706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:35.495674 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt"] Oct 30 18:32:36.033627 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:36.033595 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:36.455695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:36.455655 5624 generic.go:356] "Generic (PLEG): container finished" podID="6406ed84-abfd-457a-be38-7354061379c6" containerID="e785aad926700be9720b30627c680017ac59c76253787614a1d812d7b9508aad" exitCode=0 Oct 30 18:32:36.455848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:36.455700 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" event={"ID":"6406ed84-abfd-457a-be38-7354061379c6","Type":"ContainerDied","Data":"e785aad926700be9720b30627c680017ac59c76253787614a1d812d7b9508aad"} Oct 30 18:32:36.455848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:36.455779 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" event={"ID":"6406ed84-abfd-457a-be38-7354061379c6","Type":"ContainerStarted","Data":"1047ca39c9c82a25b8b75139897f7bb191c0b078df0738f86dd6d3f06d8823c7"} Oct 30 18:32:37.619687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.619662 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:37.657093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.657069 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6406ed84-abfd-457a-be38-7354061379c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6406ed84-abfd-457a-be38-7354061379c6-maas-api-openshift-ca\") pod \"6406ed84-abfd-457a-be38-7354061379c6\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " Oct 30 18:32:37.657254 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.657233 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6406ed84-abfd-457a-be38-7354061379c6-kube-api-access-dvvgl\" (UniqueName: \"kubernetes.io/projected/6406ed84-abfd-457a-be38-7354061379c6-kube-api-access-dvvgl\") pod \"6406ed84-abfd-457a-be38-7354061379c6\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " Oct 30 18:32:37.657309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.657274 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6406ed84-abfd-457a-be38-7354061379c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6406ed84-abfd-457a-be38-7354061379c6-maas-api-xks-ca\") pod \"6406ed84-abfd-457a-be38-7354061379c6\" (UID: \"6406ed84-abfd-457a-be38-7354061379c6\") " Oct 30 18:32:37.657463 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.657434 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6406ed84-abfd-457a-be38-7354061379c6-maas-api-openshift-ca" pod "6406ed84-abfd-457a-be38-7354061379c6" (UID: "6406ed84-abfd-457a-be38-7354061379c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:37.658007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.657982 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6406ed84-abfd-457a-be38-7354061379c6-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:37.659389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.659334 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6406ed84-abfd-457a-be38-7354061379c6-maas-api-xks-ca" pod "6406ed84-abfd-457a-be38-7354061379c6" (UID: "6406ed84-abfd-457a-be38-7354061379c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:37.659515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.659393 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6406ed84-abfd-457a-be38-7354061379c6-kube-api-access-dvvgl" pod "6406ed84-abfd-457a-be38-7354061379c6" (UID: "6406ed84-abfd-457a-be38-7354061379c6"). InnerVolumeSpecName "kube-api-access-dvvgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:37.759323 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.759248 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvvgl\" (UniqueName: \"kubernetes.io/projected/6406ed84-abfd-457a-be38-7354061379c6-kube-api-access-dvvgl\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:37.759323 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.759274 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6406ed84-abfd-457a-be38-7354061379c6-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:37.834451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:37.834416 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:38.471225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:38.471197 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" Oct 30 18:32:38.471407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:38.471235 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt" event={"ID":"6406ed84-abfd-457a-be38-7354061379c6","Type":"ContainerDied","Data":"1047ca39c9c82a25b8b75139897f7bb191c0b078df0738f86dd6d3f06d8823c7"} Oct 30 18:32:38.471407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:38.471284 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1047ca39c9c82a25b8b75139897f7bb191c0b078df0738f86dd6d3f06d8823c7" Oct 30 18:32:38.966668 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:38.966634 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:39.066987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:39.066924 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:39.307516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:39.307432 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj"] Oct 30 18:32:39.307691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:39.307653 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" containerID="cri-o://ca5f6f6cc3276c279017e32cf3f78bb471070b4fd581575f8b06aca6830780bd" gracePeriod=30 Oct 30 18:32:39.320919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:39.320891 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:39.524638 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:39.524604 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:40.982379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:40.982343 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:32:41.303075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:41.302991 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:42.820389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:42.820356 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:42.972002 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:42.971974 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:44.519320 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.519285 5624 generic.go:356] "Generic (PLEG): container finished" podID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerID="ca5f6f6cc3276c279017e32cf3f78bb471070b4fd581575f8b06aca6830780bd" exitCode=0 Oct 30 18:32:44.519696 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.519314 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" event={"ID":"11579cbd-3b9f-4f8f-865f-2d46cb24901d","Type":"ContainerDied","Data":"ca5f6f6cc3276c279017e32cf3f78bb471070b4fd581575f8b06aca6830780bd"} Oct 30 18:32:44.596367 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.596297 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:32:44.635153 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.635112 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:44.741749 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.741708 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istiod-ca-cert\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.741984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.741766 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-podinfo\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.741984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.741814 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-data\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.742189 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742161 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-data" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.742261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742195 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istiod-ca-cert" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:44.742302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742282 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-certs\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.742359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742348 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-credential-socket\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.742425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742379 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-kube-api-access-8pdx8\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-kube-api-access-8pdx8\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.742425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742406 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-token\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-token\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.742591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742547 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-certs" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742616 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-credential-socket" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742554 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-socket\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742772 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-envoy\") pod \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\" (UID: \"11579cbd-3b9f-4f8f-865f-2d46cb24901d\") " Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.742791 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-socket" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.743303 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.743334 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.743342 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.743350 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.743385 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.743361 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.744566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.744526 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-podinfo" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:32:44.744705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.744680 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-kube-api-access-8pdx8" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "kube-api-access-8pdx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:44.744840 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.744818 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-envoy" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.745115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.745101 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-token" pod "11579cbd-3b9f-4f8f-865f-2d46cb24901d" (UID: "11579cbd-3b9f-4f8f-865f-2d46cb24901d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:44.844735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.844706 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.844735 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.844736 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pdx8\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-kube-api-access-8pdx8\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.844994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.844753 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.844994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:44.844768 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11579cbd-3b9f-4f8f-865f-2d46cb24901d-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:45.055455 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.055418 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" podUID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:32:45.505091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.505050 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:45.541633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.541603 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" Oct 30 18:32:45.541633 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.541615 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj" event={"ID":"11579cbd-3b9f-4f8f-865f-2d46cb24901d","Type":"ContainerDied","Data":"a2445192326b0acbdd917cb5f45e50ceb76d25ddb2f6e88da918dd89e4584db0"} Oct 30 18:32:45.542143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.541713 5624 scope.go:122] "RemoveContainer" containerID="ca5f6f6cc3276c279017e32cf3f78bb471070b4fd581575f8b06aca6830780bd" Oct 30 18:32:45.624323 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.624274 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:45.627281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.627252 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj"] Oct 30 18:32:45.630703 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.630680 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-l2pmj"] Oct 30 18:32:45.792798 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:45.792730 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" path="/var/lib/kubelet/pods/11579cbd-3b9f-4f8f-865f-2d46cb24901d/volumes" Oct 30 18:32:46.325454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:46.325423 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:47.945079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:47.945046 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:49.445104 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:49.445057 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:49.915076 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:32:49.915033 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:32:50.345468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:50.345393 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz"] Oct 30 18:32:50.379498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:50.379471 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79"] Oct 30 18:32:50.483051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:50.483006 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:50.580372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:50.580341 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:51.041374 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:51.041345 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:52.522435 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:52.522399 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:52.936348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:52.936288 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.035770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:53.035732 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.077069 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:53.077038 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt"] Oct 30 18:32:53.089444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:53.089421 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-69c44c-s72qt"] Oct 30 18:32:53.137110 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:53.137071 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.624747 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:53.624714 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.790850 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:53.790815 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6406ed84-abfd-457a-be38-7354061379c6" path="/var/lib/kubelet/pods/6406ed84-abfd-457a-be38-7354061379c6/volumes" Oct 30 18:32:54.741682 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:54.741646 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:55.055588 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:32:55.055567 5624 logging.go:55] [core] [Channel #3296 SubChannel #3297]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.145:50052", ServerName: "10.134.0.145:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.145:50052: connect: connection refused" Oct 30 18:32:55.310239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.310213 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:32:55.345723 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.345676 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" podUID="17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" containerName="payload-processing" containerID="cri-o://f0e38528ec9596b56f86f25b047539c02623ab2427ffb8d50aa8dc621ad8f604" gracePeriod=25 Oct 30 18:32:55.380760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.380673 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" podUID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" containerName="payload-pre-processing" containerID="cri-o://a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7" gracePeriod=25 Oct 30 18:32:55.471902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.471872 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87290e0a-f02f-4aaf-84e8-3526e1b63c54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87290e0a-f02f-4aaf-84e8-3526e1b63c54-plugins-config-volume\") pod \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\" (UID: \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\") " Oct 30 18:32:55.472066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.472035 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87290e0a-f02f-4aaf-84e8-3526e1b63c54-kube-api-access-5qksp\" (UniqueName: \"kubernetes.io/projected/87290e0a-f02f-4aaf-84e8-3526e1b63c54-kube-api-access-5qksp\") pod \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\" (UID: \"87290e0a-f02f-4aaf-84e8-3526e1b63c54\") " Oct 30 18:32:55.472216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.472189 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87290e0a-f02f-4aaf-84e8-3526e1b63c54-plugins-config-volume" pod "87290e0a-f02f-4aaf-84e8-3526e1b63c54" (UID: "87290e0a-f02f-4aaf-84e8-3526e1b63c54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:55.472622 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.472602 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87290e0a-f02f-4aaf-84e8-3526e1b63c54-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:55.473969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.473910 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87290e0a-f02f-4aaf-84e8-3526e1b63c54-kube-api-access-5qksp" pod "87290e0a-f02f-4aaf-84e8-3526e1b63c54" (UID: "87290e0a-f02f-4aaf-84e8-3526e1b63c54"). InnerVolumeSpecName "kube-api-access-5qksp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:55.538100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.538021 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" podUID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:32:55.573541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.573511 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qksp\" (UniqueName: \"kubernetes.io/projected/87290e0a-f02f-4aaf-84e8-3526e1b63c54-kube-api-access-5qksp\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:55.611908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.611880 5624 generic.go:356] "Generic (PLEG): container finished" podID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" containerID="57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181" exitCode=0 Oct 30 18:32:55.612113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.611992 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" Oct 30 18:32:55.612177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.611988 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" event={"ID":"87290e0a-f02f-4aaf-84e8-3526e1b63c54","Type":"ContainerDied","Data":"57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181"} Oct 30 18:32:55.612265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.612242 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" event={"ID":"87290e0a-f02f-4aaf-84e8-3526e1b63c54","Type":"ContainerDied","Data":"6cdae25d33f238c9ed57f2585b0d51113be32c489e8f50e9283fc84aeae742a1"} Oct 30 18:32:55.612342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.612295 5624 scope.go:122] "RemoveContainer" containerID="57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181" Oct 30 18:32:55.614051 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.614027 5624 generic.go:356] "Generic (PLEG): container finished" podID="17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" containerID="f0e38528ec9596b56f86f25b047539c02623ab2427ffb8d50aa8dc621ad8f604" exitCode=0 Oct 30 18:32:55.614193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.614063 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" event={"ID":"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72","Type":"ContainerDied","Data":"f0e38528ec9596b56f86f25b047539c02623ab2427ffb8d50aa8dc621ad8f604"} Oct 30 18:32:55.618686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.618662 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:32:55.624174 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.624157 5624 scope.go:122] "RemoveContainer" containerID="57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181" Oct 30 18:32:55.624485 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:32:55.624466 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181\": container with ID starting with 57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181 not found: ID does not exist" containerID="57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181" Oct 30 18:32:55.624537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.624495 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181"} err="failed to get container status \"57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181\": rpc error: code = NotFound desc = could not find container \"57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181\": container with ID starting with 57069d7a77fa2c9bea85093162cda7a7877bd2b9c67b03dec8dca614d2acd181 not found: ID does not exist" Oct 30 18:32:55.667981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.667959 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r"] Oct 30 18:32:55.672560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.672541 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r"] Oct 30 18:32:55.775867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.775832 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-kube-api-access-m2h7x\" (UniqueName: \"kubernetes.io/projected/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-kube-api-access-m2h7x\") pod \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\" (UID: \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\") " Oct 30 18:32:55.776348 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.776042 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-plugins-config-volume\") pod \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\" (UID: \"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72\") " Oct 30 18:32:55.776484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.776462 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-plugins-config-volume" pod "17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" (UID: "17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:55.777902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.777883 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-kube-api-access-m2h7x" pod "17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" (UID: "17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72"). InnerVolumeSpecName "kube-api-access-m2h7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:55.790209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.790147 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" path="/var/lib/kubelet/pods/87290e0a-f02f-4aaf-84e8-3526e1b63c54/volumes" Oct 30 18:32:55.877188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.877156 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2h7x\" (UniqueName: \"kubernetes.io/projected/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-kube-api-access-m2h7x\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:55.877188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:55.877184 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:56.055342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:56.055307 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1b0848-5886f7c8bc-gjr5r" podUID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.145:50052\" within 1s: context deadline exceeded" Oct 30 18:32:56.334721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:56.334628 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:56.527817 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:56.527771 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" podUID="17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.150:50052\" within 1s: context deadline exceeded" Oct 30 18:32:56.528050 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:32:56.527871 5624 logging.go:55] [core] [Channel #3300 SubChannel #3302]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.150:50052", ServerName: "10.134.0.150:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.150:50052: operation was canceled" Oct 30 18:32:56.622971 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:56.622907 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" Oct 30 18:32:56.623131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:56.622967 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz" event={"ID":"17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72","Type":"ContainerDied","Data":"e1952acc494fba6a5c64b0787023989b742794f2e298871efbc6f8f3ada2a170"} Oct 30 18:32:56.623131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:56.623026 5624 scope.go:122] "RemoveContainer" containerID="f0e38528ec9596b56f86f25b047539c02623ab2427ffb8d50aa8dc621ad8f604" Oct 30 18:32:56.678318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:56.678287 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz"] Oct 30 18:32:56.691386 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:56.691359 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9a107-75cf5c66c6-cshsz"] Oct 30 18:32:57.270957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:57.270488 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:32:57.710733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:57.710684 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5"] Oct 30 18:32:57.711037 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:57.710997 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" podUID="b7cc05eb-a93b-4bd2-90ce-9e7846a74319" containerName="maas-api" containerID="cri-o://c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6" gracePeriod=30 Oct 30 18:32:57.766172 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:57.766141 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:57.791972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:57.791904 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" path="/var/lib/kubelet/pods/17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72/volumes" Oct 30 18:32:58.032917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.032893 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:32:58.206381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.206350 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-openshift-service-ca\") pod \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " Oct 30 18:32:58.206551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.206411 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-maas-api-tls\") pod \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " Oct 30 18:32:58.206551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.206445 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-kube-api-access-h42rb\" (UniqueName: \"kubernetes.io/projected/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-kube-api-access-h42rb\") pod \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " Oct 30 18:32:58.206551 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.206479 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-metrics-certs\") pod \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\" (UID: \"b7cc05eb-a93b-4bd2-90ce-9e7846a74319\") " Oct 30 18:32:58.206805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.206777 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-openshift-service-ca" pod "b7cc05eb-a93b-4bd2-90ce-9e7846a74319" (UID: "b7cc05eb-a93b-4bd2-90ce-9e7846a74319"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:58.207105 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.207081 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.208642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.208614 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-metrics-certs" pod "b7cc05eb-a93b-4bd2-90ce-9e7846a74319" (UID: "b7cc05eb-a93b-4bd2-90ce-9e7846a74319"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:58.208746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.208683 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-kube-api-access-h42rb" pod "b7cc05eb-a93b-4bd2-90ce-9e7846a74319" (UID: "b7cc05eb-a93b-4bd2-90ce-9e7846a74319"). InnerVolumeSpecName "kube-api-access-h42rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:58.208886 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.208868 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-maas-api-tls" pod "b7cc05eb-a93b-4bd2-90ce-9e7846a74319" (UID: "b7cc05eb-a93b-4bd2-90ce-9e7846a74319"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:58.308580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.308517 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.308580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.308544 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42rb\" (UniqueName: \"kubernetes.io/projected/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-kube-api-access-h42rb\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.308580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.308555 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc05eb-a93b-4bd2-90ce-9e7846a74319-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:32:58.369433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.369398 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:58.462929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.462899 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:58.641170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.641109 5624 generic.go:356] "Generic (PLEG): container finished" podID="b7cc05eb-a93b-4bd2-90ce-9e7846a74319" containerID="c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6" exitCode=0 Oct 30 18:32:58.641170 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.641159 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" Oct 30 18:32:58.641394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.641193 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" event={"ID":"b7cc05eb-a93b-4bd2-90ce-9e7846a74319","Type":"ContainerDied","Data":"c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6"} Oct 30 18:32:58.641394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.641250 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5" event={"ID":"b7cc05eb-a93b-4bd2-90ce-9e7846a74319","Type":"ContainerDied","Data":"ce61fd55e1429d0dd5fbd2cff6e27c5527e39c284d00fa41a13f021175417a09"} Oct 30 18:32:58.641394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.641279 5624 scope.go:122] "RemoveContainer" containerID="c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6" Oct 30 18:32:58.652028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.652006 5624 scope.go:122] "RemoveContainer" containerID="c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6" Oct 30 18:32:58.652341 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:32:58.652311 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6\": container with ID starting with c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6 not found: ID does not exist" containerID="c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6" Oct 30 18:32:58.652462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.652343 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6"} err="failed to get container status \"c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6\": rpc error: code = NotFound desc = could not find container \"c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6\": container with ID starting with c5f726686488e70b7677d1279bfe2b5cb27dda1fb4dec18dcf391b114d6cbdd6 not found: ID does not exist" Oct 30 18:32:58.700649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.700621 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5"] Oct 30 18:32:58.715118 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.715091 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9a107-5f6454df84-dsbr5"] Oct 30 18:32:58.944645 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:58.944502 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:59.792131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:32:59.792097 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7cc05eb-a93b-4bd2-90ce-9e7846a74319" path="/var/lib/kubelet/pods/b7cc05eb-a93b-4bd2-90ce-9e7846a74319/volumes" Oct 30 18:33:00.340418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:00.340385 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.440983 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:00.440950 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.544910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:00.544875 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.643043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:00.643009 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.848009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:00.847976 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn"] Oct 30 18:33:00.848452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:00.848234 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" containerID="cri-o://4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704" gracePeriod=30 Oct 30 18:33:00.870269 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:00.870237 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:02.039646 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:02.039616 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:03.760225 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:03.760195 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.345384 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:05.345351 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.482667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:05.482630 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.528177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:05.528142 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" podUID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:33:05.580477 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:05.580428 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.702171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:05.702134 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4"] Oct 30 18:33:05.702459 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:05.702409 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" containerID="cri-o://9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac" gracePeriod=30 Oct 30 18:33:05.726285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:05.726257 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:06.120977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.120921 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:33:06.125759 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.125736 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:33:06.217329 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217256 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-certs\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.217329 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217288 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-token\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-token\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.217556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217336 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-credential-socket\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.217556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217368 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-kube-api-access-hszsh\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-kube-api-access-hszsh\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.217556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217490 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-envoy\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.217738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217567 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-data\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.217738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217599 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-socket\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.217738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217620 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-certs" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:06.217738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217711 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9bf78e-2113-4935-a722-92de3a6be1b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9bf78e-2113-4935-a722-92de3a6be1b1-istiod-ca-cert\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.217996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.217740 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-podinfo\") pod \"ac9bf78e-2113-4935-a722-92de3a6be1b1\" (UID: \"ac9bf78e-2113-4935-a722-92de3a6be1b1\") " Oct 30 18:33:06.218179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.218089 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-socket" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:06.218686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.218301 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.218686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.218304 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-data" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:06.218686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.218325 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.218686 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.218321 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-credential-socket" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:06.219038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.218691 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9bf78e-2113-4935-a722-92de3a6be1b1-istiod-ca-cert" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:06.222131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.220477 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-kube-api-access-hszsh" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "kube-api-access-hszsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:06.222131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.220626 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-token" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:06.222131 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.221906 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-podinfo" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:06.222514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.222489 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-envoy" pod "ac9bf78e-2113-4935-a722-92de3a6be1b1" (UID: "ac9bf78e-2113-4935-a722-92de3a6be1b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:06.259182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.259149 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:06.320103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.320076 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.320103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.320105 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.320264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.320119 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9bf78e-2113-4935-a722-92de3a6be1b1-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.320264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.320132 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.320264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.320145 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.320264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.320158 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9bf78e-2113-4935-a722-92de3a6be1b1-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.320264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.320172 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hszsh\" (UniqueName: \"kubernetes.io/projected/ac9bf78e-2113-4935-a722-92de3a6be1b1-kube-api-access-hszsh\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:06.711806 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.711768 5624 generic.go:356] "Generic (PLEG): container finished" podID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerID="4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704" exitCode=0 Oct 30 18:33:06.712291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.711814 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" Oct 30 18:33:06.712291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.711845 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" event={"ID":"ac9bf78e-2113-4935-a722-92de3a6be1b1","Type":"ContainerDied","Data":"4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704"} Oct 30 18:33:06.712291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.711908 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn" event={"ID":"ac9bf78e-2113-4935-a722-92de3a6be1b1","Type":"ContainerDied","Data":"92eeb9031e6942d25238dc493c46c425cb3e8f68bfc27724b63abab3343f3751"} Oct 30 18:33:06.712291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.711958 5624 scope.go:122] "RemoveContainer" containerID="4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704" Oct 30 18:33:06.722407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.722390 5624 scope.go:122] "RemoveContainer" containerID="a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4" Oct 30 18:33:06.731188 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.731170 5624 scope.go:122] "RemoveContainer" containerID="4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704" Oct 30 18:33:06.731425 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:33:06.731409 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704\": container with ID starting with 4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704 not found: ID does not exist" containerID="4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704" Oct 30 18:33:06.731473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.731433 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704"} err="failed to get container status \"4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704\": rpc error: code = NotFound desc = could not find container \"4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704\": container with ID starting with 4a2c5a3e1b3689210a160e9fd0ea9b23bdb4447813d64a5fdc4cd05d7c52e704 not found: ID does not exist" Oct 30 18:33:06.731473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.731450 5624 scope.go:122] "RemoveContainer" containerID="a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4" Oct 30 18:33:06.731618 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:33:06.731601 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4\": container with ID starting with a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4 not found: ID does not exist" containerID="a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4" Oct 30 18:33:06.731673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.731620 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4"} err="failed to get container status \"a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4\": rpc error: code = NotFound desc = could not find container \"a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4\": container with ID starting with a2e0321d72a5063ca4946912eb23eb9b53f445b9f08b2924e73279d116bedea4 not found: ID does not exist" Oct 30 18:33:06.780557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.780533 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn"] Oct 30 18:33:06.785651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:06.785632 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-69c44c-openshift-default-578bd575cf-q4xzn"] Oct 30 18:33:07.400264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:07.400222 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:07.790457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:07.790375 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" path="/var/lib/kubelet/pods/ac9bf78e-2113-4935-a722-92de3a6be1b1/volumes" Oct 30 18:33:09.057191 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:09.057147 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:09.294592 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:09.294556 5624 patch_prober.go:29] interesting pod/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:33:09.294777 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:09.294642 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:33:10.641727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:10.641690 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:11.242496 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.242468 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d"] Oct 30 18:33:11.329980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.329901 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:11.397656 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.397629 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:33:11.474840 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.474806 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-kube-api-access-xfvwj\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-kube-api-access-xfvwj\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.474852 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-podinfo\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.474908 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-data\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.474960 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-envoy\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.474999 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-token\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-token\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475029 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-credential-socket\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475121 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-socket\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475169 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-certs\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475218 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a49bec-3cdd-4178-be33-f8a0194365d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a49bec-3cdd-4178-be33-f8a0194365d0-istiod-ca-cert\") pod \"c2a49bec-3cdd-4178-be33-f8a0194365d0\" (UID: \"c2a49bec-3cdd-4178-be33-f8a0194365d0\") " Oct 30 18:33:11.475596 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475513 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-credential-socket" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:11.475732 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475704 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-socket" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:11.475814 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475791 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.475921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475893 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a49bec-3cdd-4178-be33-f8a0194365d0-istiod-ca-cert" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:11.475921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475903 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-certs" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:11.476146 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.475997 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-data" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:11.477401 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.477378 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-podinfo" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:11.477602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.477580 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-envoy" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:11.477706 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.477614 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-token" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:11.478054 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.478032 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-kube-api-access-xfvwj" pod "c2a49bec-3cdd-4178-be33-f8a0194365d0" (UID: "c2a49bec-3cdd-4178-be33-f8a0194365d0"). InnerVolumeSpecName "kube-api-access-xfvwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:11.576467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.576438 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.576467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.576464 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.576467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.576473 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a49bec-3cdd-4178-be33-f8a0194365d0-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.576688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.576482 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfvwj\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-kube-api-access-xfvwj\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.576688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.576492 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.576688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.576501 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.576688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.576508 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.576688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.576517 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a49bec-3cdd-4178-be33-f8a0194365d0-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.752436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.752355 5624 generic.go:356] "Generic (PLEG): container finished" podID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerID="9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac" exitCode=0 Oct 30 18:33:11.752436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.752398 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" event={"ID":"c2a49bec-3cdd-4178-be33-f8a0194365d0","Type":"ContainerDied","Data":"9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac"} Oct 30 18:33:11.752436 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.752418 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" Oct 30 18:33:11.753079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.752441 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4" event={"ID":"c2a49bec-3cdd-4178-be33-f8a0194365d0","Type":"ContainerDied","Data":"66ca6ff45c646dc03a863aa877828e454b7864f3116547c9f462cd8a7fa305e1"} Oct 30 18:33:11.753079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.752470 5624 scope.go:122] "RemoveContainer" containerID="9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac" Oct 30 18:33:11.763427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.763411 5624 scope.go:122] "RemoveContainer" containerID="b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f" Oct 30 18:33:11.772580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.772553 5624 scope.go:122] "RemoveContainer" containerID="9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac" Oct 30 18:33:11.772873 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:33:11.772856 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac\": container with ID starting with 9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac not found: ID does not exist" containerID="9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac" Oct 30 18:33:11.772929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.772880 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac"} err="failed to get container status \"9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac\": rpc error: code = NotFound desc = could not find container \"9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac\": container with ID starting with 9f578b25eb226f542742f7631338645af95b29adc64d671521a6ddd133d8c8ac not found: ID does not exist" Oct 30 18:33:11.772929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.772895 5624 scope.go:122] "RemoveContainer" containerID="b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f" Oct 30 18:33:11.773232 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:33:11.773212 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f\": container with ID starting with b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f not found: ID does not exist" containerID="b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f" Oct 30 18:33:11.773326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.773235 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f"} err="failed to get container status \"b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f\": rpc error: code = NotFound desc = could not find container \"b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f\": container with ID starting with b1cf3c03a18c91847a6970467e356af2721c20e37a1cbfb001ecc7bbb56adf0f not found: ID does not exist" Oct 30 18:33:11.808379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.808354 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4"] Oct 30 18:33:11.820639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:11.820612 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c9a107-openshift-default-fdbc7864c-l7tr4"] Oct 30 18:33:12.091733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:12.091647 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:12.191803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:12.191764 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:12.288276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:12.288240 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:13.791840 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:13.791801 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" path="/var/lib/kubelet/pods/c2a49bec-3cdd-4178-be33-f8a0194365d0/volumes" Oct 30 18:33:13.827869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:13.827837 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:15.420136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.420096 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:15.525902 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:33:15.525877 5624 logging.go:55] [core] [Channel #3318 SubChannel #3319]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.149:50052", ServerName: "10.134.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.149:50052: connect: connection refused" Oct 30 18:33:15.649785 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.649726 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:33:15.726354 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.726324 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-kube-api-access-5jdhh\" (UniqueName: \"kubernetes.io/projected/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-kube-api-access-5jdhh\") pod \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\" (UID: \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\") " Oct 30 18:33:15.726530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.726452 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-plugins-config-volume\") pod \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\" (UID: \"48ddf336-41e7-4c70-9cd0-0aa2446ab35b\") " Oct 30 18:33:15.726863 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.726839 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-plugins-config-volume" pod "48ddf336-41e7-4c70-9cd0-0aa2446ab35b" (UID: "48ddf336-41e7-4c70-9cd0-0aa2446ab35b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:15.726964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.726919 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:15.728467 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.728444 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-kube-api-access-5jdhh" pod "48ddf336-41e7-4c70-9cd0-0aa2446ab35b" (UID: "48ddf336-41e7-4c70-9cd0-0aa2446ab35b"). InnerVolumeSpecName "kube-api-access-5jdhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:15.780123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.780094 5624 generic.go:356] "Generic (PLEG): container finished" podID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" containerID="a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7" exitCode=0 Oct 30 18:33:15.780245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.780145 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" Oct 30 18:33:15.780245 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.780176 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" event={"ID":"48ddf336-41e7-4c70-9cd0-0aa2446ab35b","Type":"ContainerDied","Data":"a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7"} Oct 30 18:33:15.780349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.780258 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" event={"ID":"48ddf336-41e7-4c70-9cd0-0aa2446ab35b","Type":"ContainerDied","Data":"686f2bd2c8100854c423da84669b7b4b4c444b8b4222b8cbad468f79fd37a24c"} Oct 30 18:33:15.780349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.780281 5624 scope.go:122] "RemoveContainer" containerID="a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7" Oct 30 18:33:15.790655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.790640 5624 scope.go:122] "RemoveContainer" containerID="a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7" Oct 30 18:33:15.791032 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:33:15.791006 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7\": container with ID starting with a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7 not found: ID does not exist" containerID="a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7" Oct 30 18:33:15.791097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.791043 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7"} err="failed to get container status \"a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7\": rpc error: code = NotFound desc = could not find container \"a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7\": container with ID starting with a7eaae3dc7b9cb28c463abe292fec9abba6d45a3580a216e588861b8a7dd7df7 not found: ID does not exist" Oct 30 18:33:15.827803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.827777 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdhh\" (UniqueName: \"kubernetes.io/projected/48ddf336-41e7-4c70-9cd0-0aa2446ab35b-kube-api-access-5jdhh\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:15.838088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.838048 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79"] Oct 30 18:33:15.841545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:15.841521 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79"] Oct 30 18:33:16.525840 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:16.525800 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9a107-849d99ff4c-rpc79" podUID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.149:50052\" within 1s: context deadline exceeded" Oct 30 18:33:17.037423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:17.037391 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.136490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:17.136454 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.789746 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:17.789712 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" path="/var/lib/kubelet/pods/48ddf336-41e7-4c70-9cd0-0aa2446ab35b/volumes" Oct 30 18:33:18.388355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:18.388308 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:18.860841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:18.860722 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:18.966126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:18.966079 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:19.065126 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:19.065089 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:19.541647 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:19.541615 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:19.915617 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:33:19.915581 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:33:20.634236 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:20.634201 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:22.285975 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:22.285918 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:22.583668 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:22.583578 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:33:23.934157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:23.934077 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:25.545594 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:25.545557 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.176293 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.176260 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.263923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.263873 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" containerID="cri-o://ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12" gracePeriod=45 Oct 30 18:33:26.276699 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.276667 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.356925 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.356887 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused" Oct 30 18:33:26.516482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.516432 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.546988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.546966 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:33:26.647875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.646850 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-model-cache\") pod \"d19508c5-5ea0-4090-85b3-db567fa6dd86\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " Oct 30 18:33:26.647875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.646894 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-tmp-dir\") pod \"d19508c5-5ea0-4090-85b3-db567fa6dd86\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " Oct 30 18:33:26.647875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.646998 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d19508c5-5ea0-4090-85b3-db567fa6dd86-tls-certs\" (UniqueName: \"kubernetes.io/secret/d19508c5-5ea0-4090-85b3-db567fa6dd86-tls-certs\") pod \"d19508c5-5ea0-4090-85b3-db567fa6dd86\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " Oct 30 18:33:26.647875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.647031 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19508c5-5ea0-4090-85b3-db567fa6dd86-kube-api-access-slfnf\" (UniqueName: \"kubernetes.io/projected/d19508c5-5ea0-4090-85b3-db567fa6dd86-kube-api-access-slfnf\") pod \"d19508c5-5ea0-4090-85b3-db567fa6dd86\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " Oct 30 18:33:26.647875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.647068 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-dshm\") pod \"d19508c5-5ea0-4090-85b3-db567fa6dd86\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " Oct 30 18:33:26.647875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.647140 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-home\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-home\") pod \"d19508c5-5ea0-4090-85b3-db567fa6dd86\" (UID: \"d19508c5-5ea0-4090-85b3-db567fa6dd86\") " Oct 30 18:33:26.648445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.648411 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-home" pod "d19508c5-5ea0-4090-85b3-db567fa6dd86" (UID: "d19508c5-5ea0-4090-85b3-db567fa6dd86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.648665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.648645 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-model-cache" pod "d19508c5-5ea0-4090-85b3-db567fa6dd86" (UID: "d19508c5-5ea0-4090-85b3-db567fa6dd86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.649490 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.649467 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-tmp-dir" pod "d19508c5-5ea0-4090-85b3-db567fa6dd86" (UID: "d19508c5-5ea0-4090-85b3-db567fa6dd86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.654393 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.654357 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d19508c5-5ea0-4090-85b3-db567fa6dd86-tls-certs" pod "d19508c5-5ea0-4090-85b3-db567fa6dd86" (UID: "d19508c5-5ea0-4090-85b3-db567fa6dd86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:26.656381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.656350 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19508c5-5ea0-4090-85b3-db567fa6dd86-kube-api-access-slfnf" pod "d19508c5-5ea0-4090-85b3-db567fa6dd86" (UID: "d19508c5-5ea0-4090-85b3-db567fa6dd86"). InnerVolumeSpecName "kube-api-access-slfnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:26.659001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.658977 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-dshm" pod "d19508c5-5ea0-4090-85b3-db567fa6dd86" (UID: "d19508c5-5ea0-4090-85b3-db567fa6dd86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.749209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.749128 5624 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d19508c5-5ea0-4090-85b3-db567fa6dd86-tls-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.749209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.749158 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfnf\" (UniqueName: \"kubernetes.io/projected/d19508c5-5ea0-4090-85b3-db567fa6dd86-kube-api-access-slfnf\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.749209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.749168 5624 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-dshm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.749209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.749180 5624 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-home\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.749209 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.749192 5624 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-model-cache\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.749591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.749220 5624 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d19508c5-5ea0-4090-85b3-db567fa6dd86-tmp-dir\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.853409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.853370 5624 generic.go:356] "Generic (PLEG): container finished" podID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerID="ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12" exitCode=0 Oct 30 18:33:26.853554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.853421 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" Oct 30 18:33:26.853554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.853452 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" event={"ID":"d19508c5-5ea0-4090-85b3-db567fa6dd86","Type":"ContainerDied","Data":"ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12"} Oct 30 18:33:26.853554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.853500 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d" event={"ID":"d19508c5-5ea0-4090-85b3-db567fa6dd86","Type":"ContainerDied","Data":"7a19dbd3906a46d2f87436c237d460d594ff9123a9a348afab16104d5b91355c"} Oct 30 18:33:26.853554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.853520 5624 scope.go:122] "RemoveContainer" containerID="ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12" Oct 30 18:33:26.863802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.863786 5624 scope.go:122] "RemoveContainer" containerID="ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12" Oct 30 18:33:26.864192 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:33:26.864172 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12\": container with ID starting with ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12 not found: ID does not exist" containerID="ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12" Oct 30 18:33:26.864260 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.864200 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12"} err="failed to get container status \"ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12\": rpc error: code = NotFound desc = could not find container \"ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12\": container with ID starting with ef85cb91d03921f9c4c60e1046d0f56bf5378132bd01ce8868af7358a77d8f12 not found: ID does not exist" Oct 30 18:33:26.906748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.906720 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d"] Oct 30 18:33:26.909978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:26.909959 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/premium-w2-fde2c5-kserve-554b5bc78c-lt54d"] Oct 30 18:33:27.157712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:27.157677 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:27.790695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:27.790666 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" path="/var/lib/kubelet/pods/d19508c5-5ea0-4090-85b3-db567fa6dd86/volumes" Oct 30 18:33:28.722736 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:28.722701 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:28.823237 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:28.823196 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:28.952176 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:28.952146 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.590969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:30.590897 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:32.284912 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:32.284869 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:33.838004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:33.837928 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:35.457667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:35.457634 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:36.148233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:36.148198 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:33:37.039142 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:37.039094 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:37.715700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:37.715639 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:37.812380 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:37.812325 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:38.725233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:38.725160 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.084666 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.084503 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.428825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.420715 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9"] Oct 30 18:33:40.449843 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.449743 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" podUID="8fef5526-096c-45de-a569-4e80570d9b28" containerName="maas-api" containerID="cri-o://12cd149cb0e8c60f6acd70ea41fa2152bc05a67eabc821da31e0c30195906880" gracePeriod=30 Oct 30 18:33:40.468115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.467847 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.713295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.711192 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.803626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.803405 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.971752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.971589 5624 generic.go:356] "Generic (PLEG): container finished" podID="8fef5526-096c-45de-a569-4e80570d9b28" containerID="12cd149cb0e8c60f6acd70ea41fa2152bc05a67eabc821da31e0c30195906880" exitCode=0 Oct 30 18:33:40.971752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.971669 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" event={"ID":"8fef5526-096c-45de-a569-4e80570d9b28","Type":"ContainerDied","Data":"12cd149cb0e8c60f6acd70ea41fa2152bc05a67eabc821da31e0c30195906880"} Oct 30 18:33:40.972166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.971769 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" event={"ID":"8fef5526-096c-45de-a569-4e80570d9b28","Type":"ContainerDied","Data":"eb731e1108bfb49d91497255711ba1c7f4efd7532226986148949ace568e8fde"} Oct 30 18:33:40.972166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.971794 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb731e1108bfb49d91497255711ba1c7f4efd7532226986148949ace568e8fde" Oct 30 18:33:40.985495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:40.985416 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:33:41.045790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.045704 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fef5526-096c-45de-a569-4e80570d9b28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fef5526-096c-45de-a569-4e80570d9b28-openshift-service-ca\") pod \"8fef5526-096c-45de-a569-4e80570d9b28\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " Oct 30 18:33:41.046180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.045857 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-maas-api-tls\") pod \"8fef5526-096c-45de-a569-4e80570d9b28\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " Oct 30 18:33:41.046316 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.046199 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fef5526-096c-45de-a569-4e80570d9b28-kube-api-access-r6bnj\" (UniqueName: \"kubernetes.io/projected/8fef5526-096c-45de-a569-4e80570d9b28-kube-api-access-r6bnj\") pod \"8fef5526-096c-45de-a569-4e80570d9b28\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " Oct 30 18:33:41.046558 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.046512 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fef5526-096c-45de-a569-4e80570d9b28-openshift-service-ca" pod "8fef5526-096c-45de-a569-4e80570d9b28" (UID: "8fef5526-096c-45de-a569-4e80570d9b28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:41.047190 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.047152 5624 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fef5526-096c-45de-a569-4e80570d9b28-openshift-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:41.050700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.050608 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-maas-api-tls" pod "8fef5526-096c-45de-a569-4e80570d9b28" (UID: "8fef5526-096c-45de-a569-4e80570d9b28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:41.051207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.051164 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fef5526-096c-45de-a569-4e80570d9b28-kube-api-access-r6bnj" pod "8fef5526-096c-45de-a569-4e80570d9b28" (UID: "8fef5526-096c-45de-a569-4e80570d9b28"). InnerVolumeSpecName "kube-api-access-r6bnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:41.148058 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.147948 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-metrics-certs\") pod \"8fef5526-096c-45de-a569-4e80570d9b28\" (UID: \"8fef5526-096c-45de-a569-4e80570d9b28\") " Oct 30 18:33:41.157507 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.150117 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6bnj\" (UniqueName: \"kubernetes.io/projected/8fef5526-096c-45de-a569-4e80570d9b28-kube-api-access-r6bnj\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:41.157507 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.150174 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-maas-api-tls\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:41.169515 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.169423 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-metrics-certs" pod "8fef5526-096c-45de-a569-4e80570d9b28" (UID: "8fef5526-096c-45de-a569-4e80570d9b28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:41.217625 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.217539 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:41.252103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.251890 5624 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fef5526-096c-45de-a569-4e80570d9b28-metrics-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:33:41.322660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.322574 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:41.418379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.418301 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:41.868293 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.868225 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:41.981715 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:41.981622 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9" Oct 30 18:33:42.030864 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:42.030782 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt"] Oct 30 18:33:42.039056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:42.038969 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0206e564-vb9pt"] Oct 30 18:33:42.046001 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:42.045904 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9"] Oct 30 18:33:42.050988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:42.050881 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-fde2c5-548b66898-t4sj9"] Oct 30 18:33:42.950100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:42.950026 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:43.294776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:43.294584 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:43.795923 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:43.795820 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fef5526-096c-45de-a569-4e80570d9b28" path="/var/lib/kubelet/pods/8fef5526-096c-45de-a569-4e80570d9b28/volumes" Oct 30 18:33:43.797025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:43.796955 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c29e32f-861c-48c3-946b-5ffa93867689" path="/var/lib/kubelet/pods/9c29e32f-861c-48c3-946b-5ffa93867689/volumes" Oct 30 18:33:44.936431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:44.936370 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:46.433197 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:46.433151 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:47.619887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:47.619826 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:47.719617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:47.719539 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:47.995673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:47.995589 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:48.125869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:48.125790 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:33:48.196144 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:48.196080 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:48.505449 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:48.505375 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:49.313390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:49.313305 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:49.404767 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:49.404704 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:49.656134 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:49.656056 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:49.942951 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:33:49.942762 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:33:49.969193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:49.969140 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:50.681614 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:50.681545 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:50.791893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:50.791819 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:50.885391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:50.885320 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:51.630557 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:51.630481 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:53.226921 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:53.226840 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:53.482725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:53.482541 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj"] Oct 30 18:33:53.564344 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:53.564255 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:54.835440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:54.835362 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:55.008129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:55.007882 5624 scope.go:122] "RemoveContainer" containerID="12cd149cb0e8c60f6acd70ea41fa2152bc05a67eabc821da31e0c30195906880" Oct 30 18:33:56.344495 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:56.344429 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:56.451750 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:56.451677 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:57.934269 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:57.934188 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:58.207532 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:58.207349 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:59.282290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.282223 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq"] Oct 30 18:33:59.345794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.345709 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh"] Oct 30 18:33:59.347022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.346969 5624 cpu_manager.go:414] "containerMap: removing container" podUID="17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" containerName="payload-processing" Oct 30 18:33:59.347022 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347025 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" containerName="payload-processing" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347060 5624 cpu_manager.go:414] "containerMap: removing container" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347070 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347085 5624 cpu_manager.go:414] "containerMap: removing container" podUID="b7cc05eb-a93b-4bd2-90ce-9e7846a74319" containerName="maas-api" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347095 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7cc05eb-a93b-4bd2-90ce-9e7846a74319" containerName="maas-api" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347119 5624 cpu_manager.go:414] "containerMap: removing container" podUID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" containerName="payload-pre-processing" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347128 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" containerName="payload-pre-processing" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347146 5624 cpu_manager.go:414] "containerMap: removing container" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347153 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347165 5624 cpu_manager.go:414] "containerMap: removing container" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347172 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347199 5624 cpu_manager.go:414] "containerMap: removing container" podUID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" containerName="payload-pre-processing" Oct 30 18:33:59.347223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347210 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" containerName="payload-pre-processing" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347257 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8fef5526-096c-45de-a569-4e80570d9b28" containerName="maas-api" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347265 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fef5526-096c-45de-a569-4e80570d9b28" containerName="maas-api" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347282 5624 cpu_manager.go:414] "containerMap: removing container" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347290 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347314 5624 cpu_manager.go:414] "containerMap: removing container" podUID="6406ed84-abfd-457a-be38-7354061379c6" containerName="revoke-keys" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347321 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6406ed84-abfd-457a-be38-7354061379c6" containerName="revoke-keys" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347333 5624 cpu_manager.go:414] "containerMap: removing container" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347342 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347641 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347661 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347671 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="87290e0a-f02f-4aaf-84e8-3526e1b63c54" containerName="payload-pre-processing" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347689 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fef5526-096c-45de-a569-4e80570d9b28" containerName="maas-api" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347701 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19508c5-5ea0-4090-85b3-db567fa6dd86" containerName="main" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347713 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="6406ed84-abfd-457a-be38-7354061379c6" containerName="revoke-keys" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347722 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9bf78e-2113-4935-a722-92de3a6be1b1" containerName="istio-proxy" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347738 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f1aa4a-30bb-4314-8fb1-7cc7f5dd9c72" containerName="payload-processing" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347745 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="11579cbd-3b9f-4f8f-865f-2d46cb24901d" containerName="istio-proxy" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347756 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7cc05eb-a93b-4bd2-90ce-9e7846a74319" containerName="maas-api" Oct 30 18:33:59.348482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.347767 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ddf336-41e7-4c70-9cd0-0aa2446ab35b" containerName="payload-pre-processing" Oct 30 18:33:59.356261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.356189 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.369009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.368409 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7d8\" (UniqueName: \"kubernetes.io/projected/763e57b0-023c-4c6e-960e-c87448fc42f5-kube-api-access-5w7d8\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.369616 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.369557 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.370247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.370198 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.375183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.373624 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:33:59.376310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.376158 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:33:59.376663 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.376580 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh"] Oct 30 18:33:59.389016 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.388886 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:59.439981 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.439876 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:59.440440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.439999 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:33:59.471763 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.471688 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.472179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.471979 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7d8\" (UniqueName: \"kubernetes.io/projected/763e57b0-023c-4c6e-960e-c87448fc42f5-kube-api-access-5w7d8\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.472179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.472039 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.472994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.472928 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.476468 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.476399 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.486373 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.486296 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:59.496171 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.496107 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7d8\" (UniqueName: \"kubernetes.io/projected/763e57b0-023c-4c6e-960e-c87448fc42f5-kube-api-access-5w7d8\") pod \"maas-api-revoke-sub-a85fb69d-2pflh\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:33:59.704748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:33:59.704655 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:34:00.058788 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:00.058716 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh"] Oct 30 18:34:00.061357 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:34:00.061298 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763e57b0_023c_4c6e_960e_c87448fc42f5.slice/crio-7f64c9548f6a69fec7724a001056a4314e7365d0efad9524c0aa8303195a89ce WatchSource:0}: Error finding container 7f64c9548f6a69fec7724a001056a4314e7365d0efad9524c0aa8303195a89ce: Status 404 returned error can't find the container with id 7f64c9548f6a69fec7724a001056a4314e7365d0efad9524c0aa8303195a89ce Oct 30 18:34:00.200811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:00.200730 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" event={"ID":"763e57b0-023c-4c6e-960e-c87448fc42f5","Type":"ContainerStarted","Data":"7f64c9548f6a69fec7724a001056a4314e7365d0efad9524c0aa8303195a89ce"} Oct 30 18:34:01.214470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:01.214410 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" event={"ID":"763e57b0-023c-4c6e-960e-c87448fc42f5","Type":"ContainerStarted","Data":"fe91e9144a7b455be8b0739794f1da6bf7f40b2f7e87943b22bef7c7a615175f"} Oct 30 18:34:01.258310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:01.258164 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" podStartSLOduration=2.258131107 podStartE2EDuration="2.258131107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:01.255765731 +0000 UTC m=+1935.644293850" watchObservedRunningTime="2026-10-30 18:34:01.258131107 +0000 UTC m=+1935.646659225" Oct 30 18:34:02.233079 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:02.233000 5624 generic.go:356] "Generic (PLEG): container finished" podID="763e57b0-023c-4c6e-960e-c87448fc42f5" containerID="fe91e9144a7b455be8b0739794f1da6bf7f40b2f7e87943b22bef7c7a615175f" exitCode=0 Oct 30 18:34:02.234020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:02.233087 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" event={"ID":"763e57b0-023c-4c6e-960e-c87448fc42f5","Type":"ContainerDied","Data":"fe91e9144a7b455be8b0739794f1da6bf7f40b2f7e87943b22bef7c7a615175f"} Oct 30 18:34:03.484309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.484234 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:34:03.544419 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.544186 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-openshift-ca\") pod \"763e57b0-023c-4c6e-960e-c87448fc42f5\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " Oct 30 18:34:03.544754 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.544439 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763e57b0-023c-4c6e-960e-c87448fc42f5-kube-api-access-5w7d8\" (UniqueName: \"kubernetes.io/projected/763e57b0-023c-4c6e-960e-c87448fc42f5-kube-api-access-5w7d8\") pod \"763e57b0-023c-4c6e-960e-c87448fc42f5\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " Oct 30 18:34:03.544754 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.544607 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-xks-ca\") pod \"763e57b0-023c-4c6e-960e-c87448fc42f5\" (UID: \"763e57b0-023c-4c6e-960e-c87448fc42f5\") " Oct 30 18:34:03.544922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.544790 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-openshift-ca" pod "763e57b0-023c-4c6e-960e-c87448fc42f5" (UID: "763e57b0-023c-4c6e-960e-c87448fc42f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:03.545684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.545523 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:03.551043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.549408 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-xks-ca" pod "763e57b0-023c-4c6e-960e-c87448fc42f5" (UID: "763e57b0-023c-4c6e-960e-c87448fc42f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:03.553761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.552035 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763e57b0-023c-4c6e-960e-c87448fc42f5-kube-api-access-5w7d8" pod "763e57b0-023c-4c6e-960e-c87448fc42f5" (UID: "763e57b0-023c-4c6e-960e-c87448fc42f5"). InnerVolumeSpecName "kube-api-access-5w7d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:03.646780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.646697 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/763e57b0-023c-4c6e-960e-c87448fc42f5-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:03.646780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:03.646769 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7d8\" (UniqueName: \"kubernetes.io/projected/763e57b0-023c-4c6e-960e-c87448fc42f5-kube-api-access-5w7d8\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:04.257471 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:04.257404 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" Oct 30 18:34:04.257887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:04.257404 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh" event={"ID":"763e57b0-023c-4c6e-960e-c87448fc42f5","Type":"ContainerDied","Data":"7f64c9548f6a69fec7724a001056a4314e7365d0efad9524c0aa8303195a89ce"} Oct 30 18:34:04.257887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:04.257592 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f64c9548f6a69fec7724a001056a4314e7365d0efad9524c0aa8303195a89ce" Oct 30 18:34:04.283446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:04.283283 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" containerID="cri-o://3850f5f1b274436d287689ba50b5652b1312d6fa3cce98728a2aa53807833a8a" gracePeriod=25 Oct 30 18:34:08.516877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:08.516790 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" containerID="cri-o://8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4" gracePeriod=45 Oct 30 18:34:08.583318 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:08.583247 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Oct 30 18:34:08.928114 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:08.926827 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:34:09.046353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.046250 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-tmp-dir\") pod \"61ae5481-ba97-4cea-b38e-563d3b239908\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " Oct 30 18:34:09.046758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.046413 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ae5481-ba97-4cea-b38e-563d3b239908-kube-api-access-f5zdc\" (UniqueName: \"kubernetes.io/projected/61ae5481-ba97-4cea-b38e-563d3b239908-kube-api-access-f5zdc\") pod \"61ae5481-ba97-4cea-b38e-563d3b239908\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " Oct 30 18:34:09.046758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.046537 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61ae5481-ba97-4cea-b38e-563d3b239908-tls-certs\" (UniqueName: \"kubernetes.io/secret/61ae5481-ba97-4cea-b38e-563d3b239908-tls-certs\") pod \"61ae5481-ba97-4cea-b38e-563d3b239908\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " Oct 30 18:34:09.046758 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.046646 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-dshm\") pod \"61ae5481-ba97-4cea-b38e-563d3b239908\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " Oct 30 18:34:09.047102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.046806 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-home\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-home\") pod \"61ae5481-ba97-4cea-b38e-563d3b239908\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " Oct 30 18:34:09.047102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.046859 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-tmp-dir" pod "61ae5481-ba97-4cea-b38e-563d3b239908" (UID: "61ae5481-ba97-4cea-b38e-563d3b239908"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:09.047102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.046915 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-model-cache\") pod \"61ae5481-ba97-4cea-b38e-563d3b239908\" (UID: \"61ae5481-ba97-4cea-b38e-563d3b239908\") " Oct 30 18:34:09.047805 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.047747 5624 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-tmp-dir\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:09.048136 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.047787 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-home" pod "61ae5481-ba97-4cea-b38e-563d3b239908" (UID: "61ae5481-ba97-4cea-b38e-563d3b239908"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:09.048263 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.048218 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-model-cache" pod "61ae5481-ba97-4cea-b38e-563d3b239908" (UID: "61ae5481-ba97-4cea-b38e-563d3b239908"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:09.051440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.051372 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-dshm" pod "61ae5481-ba97-4cea-b38e-563d3b239908" (UID: "61ae5481-ba97-4cea-b38e-563d3b239908"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:09.051875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.051381 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ae5481-ba97-4cea-b38e-563d3b239908-kube-api-access-f5zdc" pod "61ae5481-ba97-4cea-b38e-563d3b239908" (UID: "61ae5481-ba97-4cea-b38e-563d3b239908"). InnerVolumeSpecName "kube-api-access-f5zdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:09.054559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.054473 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ae5481-ba97-4cea-b38e-563d3b239908-tls-certs" pod "61ae5481-ba97-4cea-b38e-563d3b239908" (UID: "61ae5481-ba97-4cea-b38e-563d3b239908"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:09.149819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.149726 5624 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-dshm\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:09.149819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.149808 5624 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-home\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:09.149819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.149821 5624 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61ae5481-ba97-4cea-b38e-563d3b239908-model-cache\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:09.149819 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.149844 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5zdc\" (UniqueName: \"kubernetes.io/projected/61ae5481-ba97-4cea-b38e-563d3b239908-kube-api-access-f5zdc\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:09.150408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.149860 5624 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ae5481-ba97-4cea-b38e-563d3b239908-tls-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:09.323812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.323727 5624 generic.go:356] "Generic (PLEG): container finished" podID="61ae5481-ba97-4cea-b38e-563d3b239908" containerID="8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4" exitCode=0 Oct 30 18:34:09.323812 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.323817 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" Oct 30 18:34:09.324365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.323818 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" event={"ID":"61ae5481-ba97-4cea-b38e-563d3b239908","Type":"ContainerDied","Data":"8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4"} Oct 30 18:34:09.324365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.323900 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj" event={"ID":"61ae5481-ba97-4cea-b38e-563d3b239908","Type":"ContainerDied","Data":"7b2e1ac7962ff946963b3a0ecf5e3829c2fe3287ec9e190a2890d32bdec0fdd5"} Oct 30 18:34:09.324365 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.323928 5624 scope.go:122] "RemoveContainer" containerID="8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4" Oct 30 18:34:09.346294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.346246 5624 scope.go:122] "RemoveContainer" containerID="8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4" Oct 30 18:34:09.347081 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:34:09.347024 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4\": container with ID starting with 8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4 not found: ID does not exist" containerID="8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4" Oct 30 18:34:09.347164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.347116 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4"} err="failed to get container status \"8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4\": rpc error: code = NotFound desc = could not find container \"8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4\": container with ID starting with 8b2f936295dbebf536e10669c8478fa9361cef66c24ab7a2a888382919fb4ac4 not found: ID does not exist" Oct 30 18:34:09.383382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.383271 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj"] Oct 30 18:34:09.398207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.398119 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/premium-w4-4ef6ac-kserve-58674bf4ff-fzvkj"] Oct 30 18:34:09.796379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:09.796304 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" path="/var/lib/kubelet/pods/61ae5481-ba97-4cea-b38e-563d3b239908/volumes" Oct 30 18:34:11.904826 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:11.904753 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:12.035879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:12.035808 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:34:13.880192 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:13.880129 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:34:14.526369 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:14.526294 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:15.018156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:15.018072 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:15.111811 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:15.111687 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:15.654773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:15.654698 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:18.238284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:18.238184 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2"] Oct 30 18:34:18.239072 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:18.238565 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" containerID="cri-o://e3f1aa7c53f2eedf115342ac2fc45e5b1a4938a6e17ff3c6a57225ec02f706a2" gracePeriod=30 Oct 30 18:34:18.252615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:18.252523 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7"] Oct 30 18:34:18.253135 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:18.253053 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" containerID="cri-o://7c77088cf56846869f48ac89fbd3f53282dcff24be84fd255232deff10832404" gracePeriod=30 Oct 30 18:34:18.258346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:18.258272 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7"] Oct 30 18:34:18.258761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:18.258704 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" containerID="cri-o://1820103603e1a4ecdc728e715ea386966c5cd80495306d26dd8980ab2dd2eff9" gracePeriod=30 Oct 30 18:34:19.935115 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:34:19.935031 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:34:20.206595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:20.206432 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:20.309599 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:20.309534 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:21.547623 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:21.547534 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-bb5z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:34:21.548359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:21.547689 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:34:22.036589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.036513 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:34:22.222279 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.222204 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:34:22.437602 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.437528 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:22.478271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.476057 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-vhrw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:34:22.478271 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.476305 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:34:22.548676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.548602 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:22.789164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.788992 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:22.790427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.790357 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2"] Oct 30 18:34:22.790973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:22.790881 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" containerID="cri-o://a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1" gracePeriod=30 Oct 30 18:34:23.260519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.260445 5624 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:34:23.260964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.260556 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:34:23.483888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.483795 5624 generic.go:356] "Generic (PLEG): container finished" podID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerID="7c77088cf56846869f48ac89fbd3f53282dcff24be84fd255232deff10832404" exitCode=0 Oct 30 18:34:23.483888 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.483878 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" event={"ID":"28da4ab2-eb1e-458b-93a5-c063d30768de","Type":"ContainerDied","Data":"7c77088cf56846869f48ac89fbd3f53282dcff24be84fd255232deff10832404"} Oct 30 18:34:23.484366 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.484027 5624 scope.go:122] "RemoveContainer" containerID="747691b8f4a81b18fc3f0ae5750bc015f12b7f0c30115abef0eacdd26ab86854" Oct 30 18:34:23.498089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.490109 5624 generic.go:356] "Generic (PLEG): container finished" podID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerID="1820103603e1a4ecdc728e715ea386966c5cd80495306d26dd8980ab2dd2eff9" exitCode=0 Oct 30 18:34:23.498089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.490191 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" event={"ID":"285dc7ba-c00a-408b-a32b-8f87279f2b52","Type":"ContainerDied","Data":"1820103603e1a4ecdc728e715ea386966c5cd80495306d26dd8980ab2dd2eff9"} Oct 30 18:34:23.498089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.496077 5624 generic.go:356] "Generic (PLEG): container finished" podID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerID="e3f1aa7c53f2eedf115342ac2fc45e5b1a4938a6e17ff3c6a57225ec02f706a2" exitCode=0 Oct 30 18:34:23.498089 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.496186 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" event={"ID":"21c84ac3-01a9-4a7a-b659-31d591531f66","Type":"ContainerDied","Data":"e3f1aa7c53f2eedf115342ac2fc45e5b1a4938a6e17ff3c6a57225ec02f706a2"} Oct 30 18:34:23.653417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.653356 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:34:23.738152 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.738072 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:34:23.745514 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.745440 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:34:23.810808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.810584 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28da4ab2-eb1e-458b-93a5-c063d30768de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da4ab2-eb1e-458b-93a5-c063d30768de-istiod-ca-cert\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.810808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.810681 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-socket\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.810808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.810734 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-envoy\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.810808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.810793 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-envoy\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.810862 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c84ac3-01a9-4a7a-b659-31d591531f66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c84ac3-01a9-4a7a-b659-31d591531f66-istiod-ca-cert\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.810889 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-socket\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.810946 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-podinfo\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.810985 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-kube-api-access-jbr6m\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-kube-api-access-jbr6m\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811028 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-certs\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811117 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-token\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-token\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811172 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-credential-socket\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811220 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-kube-api-access-446ts\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-kube-api-access-446ts\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811252 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-data\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811240 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28da4ab2-eb1e-458b-93a5-c063d30768de-istiod-ca-cert" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811306 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-podinfo\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811340 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-token\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-token\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811372 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-credential-socket\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811410 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-podinfo\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811455 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285dc7ba-c00a-408b-a32b-8f87279f2b52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285dc7ba-c00a-408b-a32b-8f87279f2b52-istiod-ca-cert\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.811486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811491 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-token\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-token\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.812443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811518 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-certs\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.812443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811586 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-data\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.812443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811652 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-socket\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.812443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811734 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-credential-socket\") pod \"28da4ab2-eb1e-458b-93a5-c063d30768de\" (UID: \"28da4ab2-eb1e-458b-93a5-c063d30768de\") " Oct 30 18:34:23.812443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.811882 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-envoy\") pod \"285dc7ba-c00a-408b-a32b-8f87279f2b52\" (UID: \"285dc7ba-c00a-408b-a32b-8f87279f2b52\") " Oct 30 18:34:23.812443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.812060 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-kube-api-access-hvsm5\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-kube-api-access-hvsm5\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.812443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.812124 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-certs\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.812443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.812166 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-data\") pod \"21c84ac3-01a9-4a7a-b659-31d591531f66\" (UID: \"21c84ac3-01a9-4a7a-b659-31d591531f66\") " Oct 30 18:34:23.813716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.813164 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28da4ab2-eb1e-458b-93a5-c063d30768de-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.813716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.813166 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-socket" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.813716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.813569 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-socket" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.814698 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.814192 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-credential-socket" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.814887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.814291 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-data" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.814887 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.814802 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-certs" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.816342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.815720 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c84ac3-01a9-4a7a-b659-31d591531f66-istiod-ca-cert" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:23.816342 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.816234 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-data" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.818297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.816720 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-socket" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.818297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.816855 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-credential-socket" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.818297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.817147 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-certs" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.821077 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.819384 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-envoy" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.821894 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.821778 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-credential-socket" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.823000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.822838 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-podinfo" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:34:23.823305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.823006 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-token" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:23.823305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.823113 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-podinfo" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:34:23.823305 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.823257 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-kube-api-access-446ts" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "kube-api-access-446ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:23.823737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.823693 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-envoy" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.824370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.824310 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-token" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:23.824661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.824395 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-envoy" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.824661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.824421 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-kube-api-access-jbr6m" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "kube-api-access-jbr6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:23.824661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.824606 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-data" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.824661 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.824610 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-certs" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:23.824889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.824220 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-podinfo" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:34:23.824889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.824472 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285dc7ba-c00a-408b-a32b-8f87279f2b52-istiod-ca-cert" pod "285dc7ba-c00a-408b-a32b-8f87279f2b52" (UID: "285dc7ba-c00a-408b-a32b-8f87279f2b52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:23.825192 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.825160 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-token" pod "28da4ab2-eb1e-458b-93a5-c063d30768de" (UID: "28da4ab2-eb1e-458b-93a5-c063d30768de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:23.827123 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.827055 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-kube-api-access-hvsm5" pod "21c84ac3-01a9-4a7a-b659-31d591531f66" (UID: "21c84ac3-01a9-4a7a-b659-31d591531f66"). InnerVolumeSpecName "kube-api-access-hvsm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:23.915276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915176 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915251 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915270 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-446ts\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-kube-api-access-446ts\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915286 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915306 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915323 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915336 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915354 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915368 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/285dc7ba-c00a-408b-a32b-8f87279f2b52-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915381 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915397 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915410 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915426 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915440 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915455 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915470 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvsm5\" (UniqueName: \"kubernetes.io/projected/21c84ac3-01a9-4a7a-b659-31d591531f66-kube-api-access-hvsm5\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915485 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915498 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915510 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915521 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915535 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c84ac3-01a9-4a7a-b659-31d591531f66-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915549 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c84ac3-01a9-4a7a-b659-31d591531f66-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915564 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/285dc7ba-c00a-408b-a32b-8f87279f2b52-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915580 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28da4ab2-eb1e-458b-93a5-c063d30768de-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915593 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbr6m\" (UniqueName: \"kubernetes.io/projected/285dc7ba-c00a-408b-a32b-8f87279f2b52-kube-api-access-jbr6m\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:23.915877 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:23.915607 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28da4ab2-eb1e-458b-93a5-c063d30768de-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:24.511641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.511544 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" event={"ID":"285dc7ba-c00a-408b-a32b-8f87279f2b52","Type":"ContainerDied","Data":"9fec8ee56ff5b82759b599cc43d18212b529a9a05d769769b5e26afd8175f5f0"} Oct 30 18:34:24.512251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.512205 5624 scope.go:122] "RemoveContainer" containerID="1820103603e1a4ecdc728e715ea386966c5cd80495306d26dd8980ab2dd2eff9" Oct 30 18:34:24.512718 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.511575 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7" Oct 30 18:34:24.521559 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.521489 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" event={"ID":"21c84ac3-01a9-4a7a-b659-31d591531f66","Type":"ContainerDied","Data":"fc9987bc69e9e7e183d083c2fc6d4bf02ff8daad3974b61d97b5338aa594e237"} Oct 30 18:34:24.522337 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.522286 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2" Oct 30 18:34:24.551567 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.551384 5624 generic.go:356] "Generic (PLEG): container finished" podID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerID="3850f5f1b274436d287689ba50b5652b1312d6fa3cce98728a2aa53807833a8a" exitCode=0 Oct 30 18:34:24.553643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.552876 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" event={"ID":"5619b5d8-1160-498f-bcd3-990f8bf5def5","Type":"ContainerDied","Data":"3850f5f1b274436d287689ba50b5652b1312d6fa3cce98728a2aa53807833a8a"} Oct 30 18:34:24.562754 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.562689 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" event={"ID":"28da4ab2-eb1e-458b-93a5-c063d30768de","Type":"ContainerDied","Data":"4ae01d4b87285d7aafcb4504f3b064eb2cebfa5c15ca2201da02edcabb920309"} Oct 30 18:34:24.563408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.563296 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" Oct 30 18:34:24.565308 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.565255 5624 scope.go:122] "RemoveContainer" containerID="e3f1aa7c53f2eedf115342ac2fc45e5b1a4938a6e17ff3c6a57225ec02f706a2" Oct 30 18:34:24.590736 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.589872 5624 scope.go:122] "RemoveContainer" containerID="7c77088cf56846869f48ac89fbd3f53282dcff24be84fd255232deff10832404" Oct 30 18:34:24.614193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.614130 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7"] Oct 30 18:34:24.617354 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.617108 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-vhrw7"] Oct 30 18:34:24.658303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.658229 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7"] Oct 30 18:34:24.663992 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.663903 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:34:24.666451 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.666374 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7"] Oct 30 18:34:24.702066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.701970 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2"] Oct 30 18:34:24.706860 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.706756 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-bb5z2"] Oct 30 18:34:24.720112 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.720000 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-b9qj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:34:24.720485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.720186 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-b9qj7" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:34:24.845284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.845093 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5619b5d8-1160-498f-bcd3-990f8bf5def5-kube-api-access-htnms\" (UniqueName: \"kubernetes.io/projected/5619b5d8-1160-498f-bcd3-990f8bf5def5-kube-api-access-htnms\") pod \"5619b5d8-1160-498f-bcd3-990f8bf5def5\" (UID: \"5619b5d8-1160-498f-bcd3-990f8bf5def5\") " Oct 30 18:34:24.845284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.845202 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5619b5d8-1160-498f-bcd3-990f8bf5def5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619b5d8-1160-498f-bcd3-990f8bf5def5-plugins-config-volume\") pod \"5619b5d8-1160-498f-bcd3-990f8bf5def5\" (UID: \"5619b5d8-1160-498f-bcd3-990f8bf5def5\") " Oct 30 18:34:24.845753 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.845716 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5619b5d8-1160-498f-bcd3-990f8bf5def5-plugins-config-volume" pod "5619b5d8-1160-498f-bcd3-990f8bf5def5" (UID: "5619b5d8-1160-498f-bcd3-990f8bf5def5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:24.846091 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.846067 5624 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619b5d8-1160-498f-bcd3-990f8bf5def5-plugins-config-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:24.848896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.848834 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5619b5d8-1160-498f-bcd3-990f8bf5def5-kube-api-access-htnms" pod "5619b5d8-1160-498f-bcd3-990f8bf5def5" (UID: "5619b5d8-1160-498f-bcd3-990f8bf5def5"). InnerVolumeSpecName "kube-api-access-htnms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:24.947598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:24.947511 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnms\" (UniqueName: \"kubernetes.io/projected/5619b5d8-1160-498f-bcd3-990f8bf5def5-kube-api-access-htnms\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:25.581794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.581740 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" Oct 30 18:34:25.581794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.581748 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq" event={"ID":"5619b5d8-1160-498f-bcd3-990f8bf5def5","Type":"ContainerDied","Data":"a4bf0c1cd4185c017dbe880359e62246544701c6105eb55bbdd1d17984fc72e9"} Oct 30 18:34:25.582332 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.581915 5624 scope.go:122] "RemoveContainer" containerID="3850f5f1b274436d287689ba50b5652b1312d6fa3cce98728a2aa53807833a8a" Oct 30 18:34:25.642065 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.641984 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq"] Oct 30 18:34:25.655306 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.655218 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ef6ac-6999775c87-hbpbq"] Oct 30 18:34:25.794624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.794541 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" path="/var/lib/kubelet/pods/21c84ac3-01a9-4a7a-b659-31d591531f66/volumes" Oct 30 18:34:25.796092 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.796040 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" path="/var/lib/kubelet/pods/285dc7ba-c00a-408b-a32b-8f87279f2b52/volumes" Oct 30 18:34:25.797460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.797428 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" path="/var/lib/kubelet/pods/28da4ab2-eb1e-458b-93a5-c063d30768de/volumes" Oct 30 18:34:25.799395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:25.799341 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" path="/var/lib/kubelet/pods/5619b5d8-1160-498f-bcd3-990f8bf5def5/volumes" Oct 30 18:34:28.166406 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.166334 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:34:28.195922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.195840 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-credential-socket\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.195922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.195963 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-socket\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.195922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196009 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-podinfo\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.196545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196085 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-certs\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.196545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196137 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-token\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-token\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.196545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196207 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-kube-api-access-bldlt\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-kube-api-access-bldlt\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.196545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196346 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-data\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.196545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196402 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8824e58-efac-475b-8dbe-3f105d4d2f32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8824e58-efac-475b-8dbe-3f105d4d2f32-istiod-ca-cert\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.196545 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196506 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-envoy\") pod \"b8824e58-efac-475b-8dbe-3f105d4d2f32\" (UID: \"b8824e58-efac-475b-8dbe-3f105d4d2f32\") " Oct 30 18:34:28.196893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196597 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-socket" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.196893 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.196617 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-credential-socket" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.197274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.197088 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8824e58-efac-475b-8dbe-3f105d4d2f32-istiod-ca-cert" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:28.197274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.197173 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-data" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.197748 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.197686 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-certs" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.198251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.198205 5624 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-credential-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.198251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.198251 5624 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-socket\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.198575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.198273 5624 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-data\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.198575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.198290 5624 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8824e58-efac-475b-8dbe-3f105d4d2f32-istiod-ca-cert\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.200470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.200390 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-podinfo" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:34:28.200875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.200403 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-token" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:28.200875 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.200734 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-kube-api-access-bldlt" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "kube-api-access-bldlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:28.201349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.201308 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-envoy" pod "b8824e58-efac-475b-8dbe-3f105d4d2f32" (UID: "b8824e58-efac-475b-8dbe-3f105d4d2f32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:28.300106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.299986 5624 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-workload-certs\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.300106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.300079 5624 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-token\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.300106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.300097 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bldlt\" (UniqueName: \"kubernetes.io/projected/b8824e58-efac-475b-8dbe-3f105d4d2f32-kube-api-access-bldlt\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.300106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.300118 5624 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-envoy\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.300106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.300138 5624 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8824e58-efac-475b-8dbe-3f105d4d2f32-istio-podinfo\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.625903 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.625825 5624 generic.go:356] "Generic (PLEG): container finished" podID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerID="a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1" exitCode=0 Oct 30 18:34:28.626326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.625946 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" event={"ID":"b8824e58-efac-475b-8dbe-3f105d4d2f32","Type":"ContainerDied","Data":"a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1"} Oct 30 18:34:28.626326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.626086 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" event={"ID":"b8824e58-efac-475b-8dbe-3f105d4d2f32","Type":"ContainerDied","Data":"5ca0ea53b75d2490ea95c99b749b58a4ed000925b1882ce3b40beedffc3580f4"} Oct 30 18:34:28.626326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.625981 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2" Oct 30 18:34:28.626326 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.626132 5624 scope.go:122] "RemoveContainer" containerID="a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1" Oct 30 18:34:28.646388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.646343 5624 scope.go:122] "RemoveContainer" containerID="a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1" Oct 30 18:34:28.647105 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:34:28.647051 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1\": container with ID starting with a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1 not found: ID does not exist" containerID="a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1" Oct 30 18:34:28.647333 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.647123 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1"} err="failed to get container status \"a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1\": rpc error: code = NotFound desc = could not find container \"a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1\": container with ID starting with a8035c68024c745269331e6e44e0f67fda109e57cbbc93d2307dee569010d9d1 not found: ID does not exist" Oct 30 18:34:28.685883 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.685805 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2"] Oct 30 18:34:28.699569 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:28.699468 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-l9tx2"] Oct 30 18:34:29.647618 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:29.647530 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:29.748031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:29.747757 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:29.796986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:29.796865 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" path="/var/lib/kubelet/pods/b8824e58-efac-475b-8dbe-3f105d4d2f32/volumes" Oct 30 18:34:31.597429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:31.597344 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:37.066302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:37.066221 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:43.068571 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:43.068485 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:44.501036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:44.500896 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:48.338760 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.338667 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59"] Oct 30 18:34:48.339914 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.339845 5624 cpu_manager.go:414] "containerMap: removing container" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" Oct 30 18:34:48.339914 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.339902 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.339970 5624 cpu_manager.go:414] "containerMap: removing container" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.339982 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340008 5624 cpu_manager.go:414] "containerMap: removing container" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340019 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340030 5624 cpu_manager.go:414] "containerMap: removing container" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340040 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340062 5624 cpu_manager.go:414] "containerMap: removing container" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340070 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340082 5624 cpu_manager.go:414] "containerMap: removing container" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340092 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340140 5624 cpu_manager.go:414] "containerMap: removing container" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340148 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340173 5624 cpu_manager.go:414] "containerMap: removing container" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340180 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340192 5624 cpu_manager.go:414] "containerMap: removing container" podUID="763e57b0-023c-4c6e-960e-c87448fc42f5" containerName="revoke-keys" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340200 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763e57b0-023c-4c6e-960e-c87448fc42f5" containerName="revoke-keys" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340424 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a49bec-3cdd-4178-be33-f8a0194365d0" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340448 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ae5481-ba97-4cea-b38e-563d3b239908" containerName="main" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340459 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340469 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8824e58-efac-475b-8dbe-3f105d4d2f32" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340479 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="28da4ab2-eb1e-458b-93a5-c063d30768de" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340486 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="5619b5d8-1160-498f-bcd3-990f8bf5def5" containerName="payload-pre-processing" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340501 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="285dc7ba-c00a-408b-a32b-8f87279f2b52" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340515 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c84ac3-01a9-4a7a-b659-31d591531f66" containerName="istio-proxy" Oct 30 18:34:48.340508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.340526 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="763e57b0-023c-4c6e-960e-c87448fc42f5" containerName="revoke-keys" Oct 30 18:34:48.345394 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.345334 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.356070 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.355986 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr"] Oct 30 18:34:48.362286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.362169 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.372802 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.372715 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59"] Oct 30 18:34:48.391404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391262 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357250e3-7a1d-404b-be85-694272143fea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.391404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391340 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391560 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391627 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357250e3-7a1d-404b-be85-694272143fea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391669 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391707 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391753 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391789 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391823 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.391914 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.392023 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.392125 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqv9n\" (UniqueName: \"kubernetes.io/projected/357250e3-7a1d-404b-be85-694272143fea-kube-api-access-cqv9n\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.392184 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.392257 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttgg\" (UniqueName: \"kubernetes.io/projected/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-kube-api-access-4ttgg\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.392570 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.392569 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.393425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.392637 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.393425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.392894 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.393425 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.393003 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357250e3-7a1d-404b-be85-694272143fea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.401182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.401116 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr"] Oct 30 18:34:48.494433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494332 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.494433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494416 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357250e3-7a1d-404b-be85-694272143fea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.494433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494452 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494487 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494516 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494536 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494568 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494623 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494746 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494803 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqv9n\" (UniqueName: \"kubernetes.io/projected/357250e3-7a1d-404b-be85-694272143fea-kube-api-access-cqv9n\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494849 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494906 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttgg\" (UniqueName: \"kubernetes.io/projected/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-kube-api-access-4ttgg\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.494980 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495083 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495013 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495129 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495198 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357250e3-7a1d-404b-be85-694272143fea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495227 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495251 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495275 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357250e3-7a1d-404b-be85-694272143fea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495321 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495339 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495568 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.495704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495623 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.496248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495778 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.496248 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.495218 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.496383 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.496323 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.496733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.496685 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357250e3-7a1d-404b-be85-694272143fea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.496916 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.496697 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.499068 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.499006 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357250e3-7a1d-404b-be85-694272143fea-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.499327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.499246 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357250e3-7a1d-404b-be85-694272143fea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.499327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.499295 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.499408 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.499345 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.527510 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.527434 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357250e3-7a1d-404b-be85-694272143fea-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.530488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.530404 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttgg\" (UniqueName: \"kubernetes.io/projected/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-kube-api-access-4ttgg\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.533443 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.533367 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46323fd-bfd5-43e1-82c5-fd2b4766e38e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-f5zxr\" (UID: \"f46323fd-bfd5-43e1-82c5-fd2b4766e38e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:48.533773 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.533559 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqv9n\" (UniqueName: \"kubernetes.io/projected/357250e3-7a1d-404b-be85-694272143fea-kube-api-access-cqv9n\") pod \"maas-default-gateway-openshift-default-77f556d647-h2w59\" (UID: \"357250e3-7a1d-404b-be85-694272143fea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.677456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.677378 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:48.703247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:48.703174 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:49.041111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.041049 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59"] Oct 30 18:34:49.044469 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:34:49.044408 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357250e3_7a1d_404b_be85_694272143fea.slice/crio-5d077d06ffd086b37979f353f393b3f12a29e1a355f0bff2866627c3e7baa6dd WatchSource:0}: Error finding container 5d077d06ffd086b37979f353f393b3f12a29e1a355f0bff2866627c3e7baa6dd: Status 404 returned error can't find the container with id 5d077d06ffd086b37979f353f393b3f12a29e1a355f0bff2866627c3e7baa6dd Oct 30 18:34:49.049349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.049271 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:34:49.049655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.049407 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:34:49.049655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.049460 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:34:49.080178 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.079852 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr"] Oct 30 18:34:49.081443 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:34:49.081386 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46323fd_bfd5_43e1_82c5_fd2b4766e38e.slice/crio-0b20913693ff0880a339e3760518a345ece25fa83aff478425847ed9b0d04e5c WatchSource:0}: Error finding container 0b20913693ff0880a339e3760518a345ece25fa83aff478425847ed9b0d04e5c: Status 404 returned error can't find the container with id 0b20913693ff0880a339e3760518a345ece25fa83aff478425847ed9b0d04e5c Oct 30 18:34:49.086482 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.086379 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:34:49.086810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.086537 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:34:49.086810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.086575 5624 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 18:34:49.872701 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.872607 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" event={"ID":"f46323fd-bfd5-43e1-82c5-fd2b4766e38e","Type":"ContainerStarted","Data":"9b86096b9c3629175e9c0f1715503e8e8c2d714f92bef1de345542df397449d0"} Oct 30 18:34:49.873639 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.872731 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" event={"ID":"f46323fd-bfd5-43e1-82c5-fd2b4766e38e","Type":"ContainerStarted","Data":"0b20913693ff0880a339e3760518a345ece25fa83aff478425847ed9b0d04e5c"} Oct 30 18:34:49.877143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.877060 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" event={"ID":"357250e3-7a1d-404b-be85-694272143fea","Type":"ContainerStarted","Data":"3bbc80cf14afbfe7ed9c4864d9d35e1e9157a5837be9aa0748d01d61e29d8d13"} Oct 30 18:34:49.877433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.877187 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" event={"ID":"357250e3-7a1d-404b-be85-694272143fea","Type":"ContainerStarted","Data":"5d077d06ffd086b37979f353f393b3f12a29e1a355f0bff2866627c3e7baa6dd"} Oct 30 18:34:49.949968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:49.949819 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podStartSLOduration=1.949777941 podStartE2EDuration="1.949777941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:49.947808116 +0000 UTC m=+1984.336336233" watchObservedRunningTime="2026-10-30 18:34:49.949777941 +0000 UTC m=+1984.338306058" Oct 30 18:34:49.988052 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:34:49.984397 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:34:50.025865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:50.025602 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podStartSLOduration=2.025562417 podStartE2EDuration="2.025562417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:49.993035268 +0000 UTC m=+1984.381563388" watchObservedRunningTime="2026-10-30 18:34:50.025562417 +0000 UTC m=+1984.414090535" Oct 30 18:34:50.678424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:50.678337 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:34:50.681849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:50.681791 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:50.682039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:50.681894 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:50.707984 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:50.707071 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:34:50.712990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:50.711236 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:50.712990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:50.711350 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:34:51.678606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:51.678529 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:51.679353 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:51.678644 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:51.709996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:51.706357 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:51.709996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:51.706495 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:34:52.678595 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:52.678517 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:52.679357 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:52.678642 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:52.714230 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:52.714153 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:52.714843 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:52.714774 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:34:53.678506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:53.678344 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:53.678506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:53.678452 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:53.704180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:53.703892 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:53.704180 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:53.704057 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:34:54.679285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:54.679205 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:54.680030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:54.679314 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:54.709281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:54.709210 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:54.710548 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:54.710478 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:34:55.148216 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:55.147994 5624 scope.go:122] "RemoveContainer" containerID="e59884f7f31900fd47e991bb1b2b9b046a15cc63c30ca0db55e3a1768743153c" Oct 30 18:34:55.679070 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:55.678978 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:55.679489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:55.679104 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:55.709988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:55.709297 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:55.709988 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:55.709429 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:34:56.684088 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:56.683996 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:56.698003 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:56.697890 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:56.747489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:56.707305 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:56.748309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:56.748205 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:34:57.678822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:57.678722 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:57.679313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:57.678856 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:57.703972 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:57.703888 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:57.704705 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:57.704086 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:34:58.679293 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:58.679197 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 30 18:34:58.679770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:58.679326 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 30 18:34:58.704194 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:58.704115 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:34:58.705060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:34:58.704221 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 30 18:35:00.679417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:00.679327 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:00.680302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:00.679479 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:00.704270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:00.704191 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:00.704654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:00.704300 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:00.933585 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:00.933414 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:01.723109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:01.723011 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:01.724094 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:01.723142 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:01.760382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:01.760293 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:01.760780 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:01.760419 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:02.765335 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:02.765258 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:02.768419 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:02.765364 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:02.802695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:02.802581 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:02.803182 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:02.802773 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:03.447994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:03.447824 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:03.678698 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:03.678600 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:35:03.705267 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:03.703656 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:35:03.766692 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:03.766611 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:03.767462 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:03.766753 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:03.844506 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:03.844415 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:03.844999 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:03.844561 5624 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:04.074395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:04.074218 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:35:04.099512 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:04.099448 5624 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:35:04.153579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:04.153521 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" Oct 30 18:35:04.153957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:04.153673 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" Oct 30 18:35:19.937571 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:35:19.937478 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:35:43.183635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:43.183544 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:45.880454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:45.880378 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:35:49.990096 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:35:49.989857 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:35:50.219978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:50.219264 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:50.219978 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:50.219493 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:35:50.229993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:50.222128 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:35:50.229993 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:35:50.222270 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:36:06.565537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:06.565434 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:36:15.676741 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:15.676672 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:36:19.938416 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:36:19.938338 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:36:30.439415 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:30.439266 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:30.530617 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:30.530524 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:32.968382 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:32.968286 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:36:40.729261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:40.729169 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:36:40.730388 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:40.730314 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:36:43.295096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:43.295013 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:48.168529 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:36:48.168408 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:36:50.078083 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:36:50.078007 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:37:13.181796 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:13.181687 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:37:19.935304 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:37:19.935216 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:37:23.559330 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:23.559260 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:37:28.340284 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:28.340188 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:37:30.379117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:30.379048 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:31.254031 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:31.253795 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:43.475111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:43.475018 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:48.977179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:48.977103 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:37:49.951874 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:37:49.951766 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:37:55.741093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:55.725243 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:37:55.741093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:55.725361 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:37:55.746004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:37:55.744546 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:38:06.908956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:06.908845 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:08.255440 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:08.255365 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:08.355454 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:08.355391 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:14.170868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:14.170771 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:38:20.173300 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:38:20.173108 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:38:20.197042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:20.196376 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:20.197042 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:20.196506 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:20.201023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:20.199163 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:38:20.201023 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:20.199305 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:38:20.822098 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:20.822012 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:38:39.352853 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:39.352748 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:38:43.577004 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:43.575868 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:49.933694 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:38:49.933612 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:38:55.406576 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:55.406495 5624 scope.go:122] "RemoveContainer" containerID="e785aad926700be9720b30627c680017ac59c76253787614a1d812d7b9508aad" Oct 30 18:38:55.814233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:55.814037 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:55.893908 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:55.893806 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:58.557840 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:38:58.557715 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:39:04.028153 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:39:04.028042 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh"] Oct 30 18:39:04.041737 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:39:04.041646 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a85fb69d-2pflh"] Oct 30 18:39:05.793549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:39:05.793475 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763e57b0-023c-4c6e-960e-c87448fc42f5" path="/var/lib/kubelet/pods/763e57b0-023c-4c6e-960e-c87448fc42f5/volumes" Oct 30 18:39:09.371375 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:39:09.371244 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:39:11.647376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:39:11.647282 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:15.624143 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:39:15.624036 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:39:19.933133 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:39:19.933063 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:39:43.067165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:39:43.066967 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:39:43.696475 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:39:43.696381 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:49.934191 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:39:49.934110 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:40:01.885794 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:01.885673 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:40:05.206251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:05.205194 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:40:05.206251 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:05.205386 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:05.215611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:05.207306 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:40:05.215611 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:05.207431 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:12.085060 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:12.084886 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:40:19.937390 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:40:19.937314 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:40:34.737247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:34.737084 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:35.427980 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:35.427648 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:38.729965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:38.729859 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:38.834589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:38.834510 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:40.205009 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:40.204924 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:40.299336 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:40.299266 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:40.411986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:40.399545 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:40.578878 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:40.578697 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:40.675185 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:40.675107 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:41.831217 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:41.831149 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:43.327067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:43.327008 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:43.816575 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:43.816511 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:43.951771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:43.951656 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:40:44.836755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:44.836668 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:46.440448 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:46.440356 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:48.083290 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:48.083195 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:49.841117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:49.841057 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:49.955740 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:40:49.955667 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:40:51.429141 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:51.429063 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:52.916247 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:52.916179 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:53.654493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:53.654419 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:54.262996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:54.262864 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:54.368858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:54.368785 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:54.462231 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:54.462166 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:55.465224 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:55.465139 5624 scope.go:122] "RemoveContainer" containerID="fe91e9144a7b455be8b0739794f1da6bf7f40b2f7e87943b22bef7c7a615175f" Oct 30 18:40:56.070155 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:56.070093 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:56.162770 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:56.162705 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:57.930107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:57.930029 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:57.997977 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:57.997476 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:40:59.650687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:40:59.650587 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:01.220568 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:01.220491 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:02.808729 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:02.808654 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:04.316063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:04.315888 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:05.821957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:05.821874 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:07.327544 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:07.327472 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:08.844028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:08.843897 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:10.424223 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:10.424152 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:11.923889 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:11.923828 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:13.491522 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:13.491446 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:14.679996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:14.679897 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:41:15.143642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:15.143566 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:16.636636 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:16.636552 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:18.120526 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:18.120447 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:19.593762 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:19.593641 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:19.937276 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:41:19.937187 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:41:21.111663 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:21.111570 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:22.637297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:22.637218 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:24.117684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:24.117551 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:25.628338 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:25.628274 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:27.219424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:27.219355 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:28.052159 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.052074 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj"] Oct 30 18:41:28.057814 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.057760 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.076689 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.076612 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj"] Oct 30 18:41:28.082097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.082025 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:41:28.082097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.082067 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:41:28.082097 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.082065 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:41:28.087531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.087466 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:41:28.141604 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.141508 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.142120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.141850 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.142120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.141891 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmfd\" (UniqueName: \"kubernetes.io/projected/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-kube-api-access-5tmfd\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.243598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.243498 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.243598 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.243587 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmfd\" (UniqueName: \"kubernetes.io/projected/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-kube-api-access-5tmfd\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.244586 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.243649 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.245012 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.244881 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.247956 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.247883 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.265869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.265799 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmfd\" (UniqueName: \"kubernetes.io/projected/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-kube-api-access-5tmfd\") pod \"maas-api-revoke-sub-29ec5479-h9nqj\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.385390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.385305 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:28.729498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.729432 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj"] Oct 30 18:41:28.733477 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:41:28.733413 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe88d3f_2ec9_41cf_98e3_5fc7d359c45e.slice/crio-82b7cac147669ecb52ce1a40f8014ad420cecbb7ac0339921b6d04572e4ce18e WatchSource:0}: Error finding container 82b7cac147669ecb52ce1a40f8014ad420cecbb7ac0339921b6d04572e4ce18e: Status 404 returned error can't find the container with id 82b7cac147669ecb52ce1a40f8014ad420cecbb7ac0339921b6d04572e4ce18e Oct 30 18:41:28.802792 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.802609 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" event={"ID":"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e","Type":"ContainerStarted","Data":"82b7cac147669ecb52ce1a40f8014ad420cecbb7ac0339921b6d04572e4ce18e"} Oct 30 18:41:28.824281 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:28.824191 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:29.815578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:29.815511 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" event={"ID":"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e","Type":"ContainerStarted","Data":"0979e4c7e9731b8b6e7c59f4f839bff44822e528f67292467310f36dad7b3e47"} Oct 30 18:41:29.858429 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:29.858275 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" podStartSLOduration=1.858241505 podStartE2EDuration="1.858241505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:29.85610551 +0000 UTC m=+2384.244633630" watchObservedRunningTime="2026-10-30 18:41:29.858241505 +0000 UTC m=+2384.246769876" Oct 30 18:41:30.322417 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:30.322337 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:30.828460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:30.828379 5624 generic.go:356] "Generic (PLEG): container finished" podID="dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e" containerID="0979e4c7e9731b8b6e7c59f4f839bff44822e528f67292467310f36dad7b3e47" exitCode=0 Oct 30 18:41:30.829235 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:30.828484 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" event={"ID":"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e","Type":"ContainerDied","Data":"0979e4c7e9731b8b6e7c59f4f839bff44822e528f67292467310f36dad7b3e47"} Oct 30 18:41:31.827028 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:31.826918 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:32.104297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.104117 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:32.216285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.216187 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-xks-ca\") pod \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " Oct 30 18:41:32.216665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.216429 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-openshift-ca\") pod \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " Oct 30 18:41:32.216665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.216538 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-kube-api-access-5tmfd\" (UniqueName: \"kubernetes.io/projected/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-kube-api-access-5tmfd\") pod \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\" (UID: \"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e\") " Oct 30 18:41:32.216953 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.216903 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-openshift-ca" pod "dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e" (UID: "dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:32.217357 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.217293 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:41:32.220379 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.220288 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-xks-ca" pod "dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e" (UID: "dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:32.220731 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.220404 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-kube-api-access-5tmfd" pod "dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e" (UID: "dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e"). InnerVolumeSpecName "kube-api-access-5tmfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:32.318774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.318675 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmfd\" (UniqueName: \"kubernetes.io/projected/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-kube-api-access-5tmfd\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:41:32.318774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.318752 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:41:32.848870 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.848811 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" Oct 30 18:41:32.849288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.848812 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj" event={"ID":"dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e","Type":"ContainerDied","Data":"82b7cac147669ecb52ce1a40f8014ad420cecbb7ac0339921b6d04572e4ce18e"} Oct 30 18:41:32.849288 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:32.849050 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82b7cac147669ecb52ce1a40f8014ad420cecbb7ac0339921b6d04572e4ce18e" Oct 30 18:41:33.440322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:33.440251 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:34.934968 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:34.934891 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:36.517899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:36.517798 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:38.029398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:38.029291 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:39.019484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.019417 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:39.274489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.274305 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp"] Oct 30 18:41:39.275457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.275411 5624 cpu_manager.go:414] "containerMap: removing container" podUID="dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e" containerName="revoke-keys" Oct 30 18:41:39.275457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.275459 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e" containerName="revoke-keys" Oct 30 18:41:39.275606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.275593 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e" containerName="revoke-keys" Oct 30 18:41:39.280418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.280366 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.300830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.300763 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp"] Oct 30 18:41:39.305122 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.305060 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:41:39.305519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.305074 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:41:39.305519 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.305317 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:41:39.311096 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.311017 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:41:39.357905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.357811 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4ld\" (UniqueName: \"kubernetes.io/projected/8dfeb156-ec69-47de-abc3-652fa1fb7858-kube-api-access-4w4ld\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.358298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.358128 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.358298 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.358265 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.459852 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.459765 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.460310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.459925 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.460310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.460091 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4ld\" (UniqueName: \"kubernetes.io/projected/8dfeb156-ec69-47de-abc3-652fa1fb7858-kube-api-access-4w4ld\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.461164 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.461106 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.464081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.464023 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.485029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.484925 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4ld\" (UniqueName: \"kubernetes.io/projected/8dfeb156-ec69-47de-abc3-652fa1fb7858-kube-api-access-4w4ld\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.612405 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.612199 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:39.640820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.640747 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:39.984918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:39.984803 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp"] Oct 30 18:41:39.988277 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:41:39.988208 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfeb156_ec69_47de_abc3_652fa1fb7858.slice/crio-26bcca95d3646ce1ee7d9e93b4cd5b7c45f53acfe48e55f897f9d7eb58961dfe WatchSource:0}: Error finding container 26bcca95d3646ce1ee7d9e93b4cd5b7c45f53acfe48e55f897f9d7eb58961dfe: Status 404 returned error can't find the container with id 26bcca95d3646ce1ee7d9e93b4cd5b7c45f53acfe48e55f897f9d7eb58961dfe Oct 30 18:41:40.095007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:40.094884 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:41:40.928922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:40.928841 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" event={"ID":"8dfeb156-ec69-47de-abc3-652fa1fb7858","Type":"ContainerStarted","Data":"4a1258dea0b710410272190eb82e388341f175a353ba7dfcc19d62507912def2"} Oct 30 18:41:40.928922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:40.928997 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" event={"ID":"8dfeb156-ec69-47de-abc3-652fa1fb7858","Type":"ContainerStarted","Data":"26bcca95d3646ce1ee7d9e93b4cd5b7c45f53acfe48e55f897f9d7eb58961dfe"} Oct 30 18:41:40.976899 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:40.976757 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" podStartSLOduration=1.976727903 podStartE2EDuration="1.976727903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:40.97559954 +0000 UTC m=+2395.364127663" watchObservedRunningTime="2026-10-30 18:41:40.976727903 +0000 UTC m=+2395.365256023" Oct 30 18:41:41.421995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:41.399099 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:41.943221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:41.943149 5624 generic.go:356] "Generic (PLEG): container finished" podID="8dfeb156-ec69-47de-abc3-652fa1fb7858" containerID="4a1258dea0b710410272190eb82e388341f175a353ba7dfcc19d62507912def2" exitCode=0 Oct 30 18:41:41.943221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:41.943227 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" event={"ID":"8dfeb156-ec69-47de-abc3-652fa1fb7858","Type":"ContainerDied","Data":"4a1258dea0b710410272190eb82e388341f175a353ba7dfcc19d62507912def2"} Oct 30 18:41:42.944511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:42.944444 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:43.229667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.229569 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:43.337511 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.337423 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dfeb156-ec69-47de-abc3-652fa1fb7858-kube-api-access-4w4ld\" (UniqueName: \"kubernetes.io/projected/8dfeb156-ec69-47de-abc3-652fa1fb7858-kube-api-access-4w4ld\") pod \"8dfeb156-ec69-47de-abc3-652fa1fb7858\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " Oct 30 18:41:43.337910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.337642 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-openshift-ca\") pod \"8dfeb156-ec69-47de-abc3-652fa1fb7858\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " Oct 30 18:41:43.337910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.337724 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-xks-ca\") pod \"8dfeb156-ec69-47de-abc3-652fa1fb7858\" (UID: \"8dfeb156-ec69-47de-abc3-652fa1fb7858\") " Oct 30 18:41:43.338244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.338197 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-openshift-ca" pod "8dfeb156-ec69-47de-abc3-652fa1fb7858" (UID: "8dfeb156-ec69-47de-abc3-652fa1fb7858"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:43.338486 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.338460 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:41:43.341825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.341737 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-xks-ca" pod "8dfeb156-ec69-47de-abc3-652fa1fb7858" (UID: "8dfeb156-ec69-47de-abc3-652fa1fb7858"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:43.341825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.341792 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfeb156-ec69-47de-abc3-652fa1fb7858-kube-api-access-4w4ld" pod "8dfeb156-ec69-47de-abc3-652fa1fb7858" (UID: "8dfeb156-ec69-47de-abc3-652fa1fb7858"). InnerVolumeSpecName "kube-api-access-4w4ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:43.440407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.440320 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4ld\" (UniqueName: \"kubernetes.io/projected/8dfeb156-ec69-47de-abc3-652fa1fb7858-kube-api-access-4w4ld\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:41:43.440407 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.440394 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dfeb156-ec69-47de-abc3-652fa1fb7858-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:41:43.931822 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.931734 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:43.964738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.964683 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" Oct 30 18:41:43.964738 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.964680 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp" event={"ID":"8dfeb156-ec69-47de-abc3-652fa1fb7858","Type":"ContainerDied","Data":"26bcca95d3646ce1ee7d9e93b4cd5b7c45f53acfe48e55f897f9d7eb58961dfe"} Oct 30 18:41:43.965703 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:43.964801 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26bcca95d3646ce1ee7d9e93b4cd5b7c45f53acfe48e55f897f9d7eb58961dfe" Oct 30 18:41:44.089391 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:44.089192 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:44.433560 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:44.433473 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:45.918183 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:45.918108 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:47.427691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:47.427617 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:48.923995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:48.923865 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:49.884426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:49.884346 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:49.941722 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:41:49.941646 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:41:49.982848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:49.982781 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:50.470045 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:50.469926 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:51.769032 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:51.768952 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:52.231554 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:52.231475 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:52.329434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:52.329362 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:52.371652 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:52.371574 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp"] Oct 30 18:41:52.384325 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:52.384236 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f57fb7-45qrp"] Oct 30 18:41:52.905634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:52.905552 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:53.744166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:53.744077 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:41:53.799672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:53.799597 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dfeb156-ec69-47de-abc3-652fa1fb7858" path="/var/lib/kubelet/pods/8dfeb156-ec69-47de-abc3-652fa1fb7858/volumes" Oct 30 18:41:58.634721 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:58.634629 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:58.737733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:58.737649 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:59.082270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:41:59.082065 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:17.204367 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:17.204297 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:18.423821 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.423745 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g"] Oct 30 18:42:18.424658 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.424152 5624 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" containerID="cri-o://c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54" gracePeriod=10 Oct 30 18:42:18.802866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.802785 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:42:18.832111 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.832026 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4c541a-41f4-4af4-a006-3ce2155611bd-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4c541a-41f4-4af4-a006-3ce2155611bd-extensions-socket-volume\") pod \"bd4c541a-41f4-4af4-a006-3ce2155611bd\" (UID: \"bd4c541a-41f4-4af4-a006-3ce2155611bd\") " Oct 30 18:42:18.832484 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.832151 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4c541a-41f4-4af4-a006-3ce2155611bd-kube-api-access-vltdg\" (UniqueName: \"kubernetes.io/projected/bd4c541a-41f4-4af4-a006-3ce2155611bd-kube-api-access-vltdg\") pod \"bd4c541a-41f4-4af4-a006-3ce2155611bd\" (UID: \"bd4c541a-41f4-4af4-a006-3ce2155611bd\") " Oct 30 18:42:18.834043 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.833983 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4c541a-41f4-4af4-a006-3ce2155611bd-extensions-socket-volume" pod "bd4c541a-41f4-4af4-a006-3ce2155611bd" (UID: "bd4c541a-41f4-4af4-a006-3ce2155611bd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:18.837372 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.837286 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4c541a-41f4-4af4-a006-3ce2155611bd-kube-api-access-vltdg" pod "bd4c541a-41f4-4af4-a006-3ce2155611bd" (UID: "bd4c541a-41f4-4af4-a006-3ce2155611bd"). InnerVolumeSpecName "kube-api-access-vltdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:18.934691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.934622 5624 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd4c541a-41f4-4af4-a006-3ce2155611bd-extensions-socket-volume\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:42:18.934691 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:18.934683 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vltdg\" (UniqueName: \"kubernetes.io/projected/bd4c541a-41f4-4af4-a006-3ce2155611bd-kube-api-access-vltdg\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:42:19.359619 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.359508 5624 generic.go:356] "Generic (PLEG): container finished" podID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerID="c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54" exitCode=0 Oct 30 18:42:19.359619 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.359601 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" event={"ID":"bd4c541a-41f4-4af4-a006-3ce2155611bd","Type":"ContainerDied","Data":"c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54"} Oct 30 18:42:19.360274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.359630 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" Oct 30 18:42:19.360274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.359743 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g" event={"ID":"bd4c541a-41f4-4af4-a006-3ce2155611bd","Type":"ContainerDied","Data":"37c9025d584cc068e457a092796f4827d6f53e88c4f222226b12292b4db79cfc"} Oct 30 18:42:19.360274 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.359789 5624 scope.go:122] "RemoveContainer" containerID="c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54" Oct 30 18:42:19.381376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.381328 5624 scope.go:122] "RemoveContainer" containerID="695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a" Oct 30 18:42:19.406950 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.406887 5624 scope.go:122] "RemoveContainer" containerID="c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54" Oct 30 18:42:19.407681 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:42:19.407627 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54\": container with ID starting with c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54 not found: ID does not exist" containerID="c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54" Oct 30 18:42:19.407797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.407697 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54"} err="failed to get container status \"c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54\": rpc error: code = NotFound desc = could not find container \"c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54\": container with ID starting with c4d4a2c7ea4407721fcc9f9db1e7e3464716df9cb2bb57b5ae51a8c315ea4c54 not found: ID does not exist" Oct 30 18:42:19.407797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.407740 5624 scope.go:122] "RemoveContainer" containerID="695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a" Oct 30 18:42:19.408522 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:42:19.408464 5624 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a\": container with ID starting with 695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a not found: ID does not exist" containerID="695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a" Oct 30 18:42:19.408603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.408538 5624 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a"} err="failed to get container status \"695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a\": rpc error: code = NotFound desc = could not find container \"695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a\": container with ID starting with 695cf48ec4f2e5a3e1c70d8e9bcdafd38ab5d362657d4ae21d64f2d135aa596a not found: ID does not exist" Oct 30 18:42:19.418895 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.418808 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g"] Oct 30 18:42:19.423716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.423637 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x262g"] Oct 30 18:42:19.793473 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:19.793391 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" path="/var/lib/kubelet/pods/bd4c541a-41f4-4af4-a006-3ce2155611bd/volumes" Oct 30 18:42:19.935036 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:42:19.934954 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:42:20.524428 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:20.524358 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:21.153095 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:21.153019 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:26.704869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:26.704548 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:38.153190 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:38.153122 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:44.067664 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:44.067577 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:44.152695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:42:44.152638 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:49.934177 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:42:49.933987 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:43:19.934163 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:43:19.934093 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:43:21.635649 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.635542 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w"] Oct 30 18:43:21.636422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636395 5624 cpu_manager.go:414] "containerMap: removing container" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" Oct 30 18:43:21.636422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636424 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" Oct 30 18:43:21.636536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636456 5624 cpu_manager.go:414] "containerMap: removing container" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" Oct 30 18:43:21.636536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636463 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" Oct 30 18:43:21.636536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636481 5624 cpu_manager.go:414] "containerMap: removing container" podUID="8dfeb156-ec69-47de-abc3-652fa1fb7858" containerName="revoke-keys" Oct 30 18:43:21.636536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636487 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dfeb156-ec69-47de-abc3-652fa1fb7858" containerName="revoke-keys" Oct 30 18:43:21.636687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636610 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dfeb156-ec69-47de-abc3-652fa1fb7858" containerName="revoke-keys" Oct 30 18:43:21.636687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636631 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" Oct 30 18:43:21.636687 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.636641 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4c541a-41f4-4af4-a006-3ce2155611bd" containerName="manager" Oct 30 18:43:21.641483 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.641412 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:21.654885 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.654805 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w"] Oct 30 18:43:21.660530 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.660475 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-726w6\"" Oct 30 18:43:21.680759 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.680667 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9933a5a-2dec-4927-9bb8-c38905943483-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7295w\" (UID: \"a9933a5a-2dec-4927-9bb8-c38905943483\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:21.680759 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.680751 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwv2\" (UniqueName: \"kubernetes.io/projected/a9933a5a-2dec-4927-9bb8-c38905943483-kube-api-access-8zwv2\") pod \"kuadrant-operator-controller-manager-678d87766f-7295w\" (UID: \"a9933a5a-2dec-4927-9bb8-c38905943483\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:21.782307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.782219 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9933a5a-2dec-4927-9bb8-c38905943483-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7295w\" (UID: \"a9933a5a-2dec-4927-9bb8-c38905943483\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:21.782307 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.782308 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwv2\" (UniqueName: \"kubernetes.io/projected/a9933a5a-2dec-4927-9bb8-c38905943483-kube-api-access-8zwv2\") pod \"kuadrant-operator-controller-manager-678d87766f-7295w\" (UID: \"a9933a5a-2dec-4927-9bb8-c38905943483\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:21.783109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.783062 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9933a5a-2dec-4927-9bb8-c38905943483-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7295w\" (UID: \"a9933a5a-2dec-4927-9bb8-c38905943483\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:21.805457 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.805371 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwv2\" (UniqueName: \"kubernetes.io/projected/a9933a5a-2dec-4927-9bb8-c38905943483-kube-api-access-8zwv2\") pod \"kuadrant-operator-controller-manager-678d87766f-7295w\" (UID: \"a9933a5a-2dec-4927-9bb8-c38905943483\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:21.972297 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:21.972100 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:22.341635 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:22.341558 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w"] Oct 30 18:43:22.344382 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:43:22.344304 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9933a5a_2dec_4927_9bb8_c38905943483.slice/crio-0203cfe6870dced46025eda8a4b860a7f7073d49d1ef9ce72baf466cc531e48e WatchSource:0}: Error finding container 0203cfe6870dced46025eda8a4b860a7f7073d49d1ef9ce72baf466cc531e48e: Status 404 returned error can't find the container with id 0203cfe6870dced46025eda8a4b860a7f7073d49d1ef9ce72baf466cc531e48e Oct 30 18:43:22.977102 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:22.977010 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" event={"ID":"a9933a5a-2dec-4927-9bb8-c38905943483","Type":"ContainerStarted","Data":"15f7c37af63b866edd7ee5cf44bd23187a7b2863f7aa328107a1ec6b2e9756ab"} Oct 30 18:43:22.978074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:22.977158 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:22.978074 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:22.977180 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" event={"ID":"a9933a5a-2dec-4927-9bb8-c38905943483","Type":"ContainerStarted","Data":"0203cfe6870dced46025eda8a4b860a7f7073d49d1ef9ce72baf466cc531e48e"} Oct 30 18:43:23.052778 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:23.052663 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" podStartSLOduration=2.052641025 podStartE2EDuration="2.052641025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:23.03763827 +0000 UTC m=+2497.426166387" watchObservedRunningTime="2026-10-30 18:43:23.052641025 +0000 UTC m=+2497.441169141" Oct 30 18:43:27.968548 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:27.968450 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:28.048631 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:28.048563 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:33.990133 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:33.990045 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7295w" Oct 30 18:43:41.274675 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:41.274555 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:43:44.246924 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:44.246805 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:49.933325 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:43:49.933250 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:43:58.365426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:43:58.365236 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:44:04.546842 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:04.546756 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:44:05.204312 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:05.204204 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:44:05.204986 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:05.204874 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:44:05.205381 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:05.205345 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:44:05.205566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:05.205533 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:44:19.936792 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:44:19.936713 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:44:23.047253 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:23.047168 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:44:23.233918 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:23.233834 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:23.748736 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:23.748654 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:23.835196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:23.835091 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:26.979396 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:26.979286 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:27.090427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:27.090327 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:28.514044 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:28.513969 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:28.611350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:28.611249 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:28.785600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:28.785385 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:28.888130 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:28.888039 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:30.165204 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:30.165143 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:30.954437 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:30.954351 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:44:31.827258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:31.827193 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:33.347634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:33.347570 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:34.955295 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:34.955230 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:36.632642 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:36.632576 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:38.248994 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:38.248867 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:39.824421 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:39.824337 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:41.447655 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:41.447575 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:41.968620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:41.968544 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:42.655158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:42.655097 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:42.855302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:42.855220 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:42.960666 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:42.960475 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:43.164901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:43.164833 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:43.885075 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:43.884990 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:44.347160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:44.346974 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:44.761808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:44.761732 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:45.658777 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:45.658690 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:44:46.477485 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:46.477387 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:48.024313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:48.024219 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:49.525712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:49.525624 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:49.939825 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:44:49.939738 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:44:50.931591 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:50.931524 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:51.035207 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:51.035106 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:52.530309 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:52.530230 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:54.036648 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:54.036572 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:55.535370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:55.535283 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:57.030422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:57.030346 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:58.524685 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:44:58.524616 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.022039 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.021881 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.141996 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.141889 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2"] Oct 30 18:45:00.148803 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.148742 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:00.164052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.163968 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2"] Oct 30 18:45:00.171157 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.171094 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:45:00.171566 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.171164 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:45:00.171703 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.171257 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:45:00.245696 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.245609 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52j5k\" (UniqueName: \"kubernetes.io/projected/69d1f271-0c16-486a-aa84-6ca54c6877e7-kube-api-access-52j5k\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2\" (UID: \"69d1f271-0c16-486a-aa84-6ca54c6877e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:00.246167 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.246112 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d1f271-0c16-486a-aa84-6ca54c6877e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2\" (UID: \"69d1f271-0c16-486a-aa84-6ca54c6877e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:00.347531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.347337 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52j5k\" (UniqueName: \"kubernetes.io/projected/69d1f271-0c16-486a-aa84-6ca54c6877e7-kube-api-access-52j5k\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2\" (UID: \"69d1f271-0c16-486a-aa84-6ca54c6877e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:00.347919 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.347593 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d1f271-0c16-486a-aa84-6ca54c6877e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2\" (UID: \"69d1f271-0c16-486a-aa84-6ca54c6877e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:00.348995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.348909 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d1f271-0c16-486a-aa84-6ca54c6877e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2\" (UID: \"69d1f271-0c16-486a-aa84-6ca54c6877e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:00.371612 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.371531 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52j5k\" (UniqueName: \"kubernetes.io/projected/69d1f271-0c16-486a-aa84-6ca54c6877e7-kube-api-access-52j5k\") pod \"maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2\" (UID: \"69d1f271-0c16-486a-aa84-6ca54c6877e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:00.479181 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.479062 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:00.826470 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:00.826391 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2"] Oct 30 18:45:00.828949 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:45:00.828862 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d1f271_0c16_486a_aa84_6ca54c6877e7.slice/crio-aeaa3ab24c823007446b1bc14bf1c68e1470b8f894511856921615d78adf46e7 WatchSource:0}: Error finding container aeaa3ab24c823007446b1bc14bf1c68e1470b8f894511856921615d78adf46e7: Status 404 returned error can't find the container with id aeaa3ab24c823007446b1bc14bf1c68e1470b8f894511856921615d78adf46e7 Oct 30 18:45:01.214061 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:01.213909 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" event={"ID":"69d1f271-0c16-486a-aa84-6ca54c6877e7","Type":"ContainerStarted","Data":"aeaa3ab24c823007446b1bc14bf1c68e1470b8f894511856921615d78adf46e7"} Oct 30 18:45:01.619463 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:01.619266 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:02.035784 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:02.035686 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:45:02.230315 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:02.230243 5624 generic.go:356] "Generic (PLEG): container finished" podID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerID="d6606522dd52e4666ff56e35d2cb0ad04d11f347bb16f8d84b14fb003c3fbbd3" exitCode=60 Oct 30 18:45:02.231093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:02.230357 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" event={"ID":"69d1f271-0c16-486a-aa84-6ca54c6877e7","Type":"ContainerDied","Data":"d6606522dd52e4666ff56e35d2cb0ad04d11f347bb16f8d84b14fb003c3fbbd3"} Oct 30 18:45:02.231093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:02.230924 5624 scope.go:122] "RemoveContainer" containerID="d6606522dd52e4666ff56e35d2cb0ad04d11f347bb16f8d84b14fb003c3fbbd3" Oct 30 18:45:03.246020 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:03.245914 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" event={"ID":"69d1f271-0c16-486a-aa84-6ca54c6877e7","Type":"ContainerStarted","Data":"9386cf650148a242fc2393f1bb32f50b39e3c0991a2c04affa4a09442fac6d87"} Oct 30 18:45:03.293508 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:03.293335 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" podStartSLOduration=3.293289278 podStartE2EDuration="3.293289278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:03.286349964 +0000 UTC m=+2597.674878082" watchObservedRunningTime="2026-10-30 18:45:03.293289278 +0000 UTC m=+2597.681817403" Oct 30 18:45:03.329199 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:03.329106 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:04.260700 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:04.260624 5624 generic.go:356] "Generic (PLEG): container finished" podID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerID="9386cf650148a242fc2393f1bb32f50b39e3c0991a2c04affa4a09442fac6d87" exitCode=60 Oct 30 18:45:04.261744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:04.260733 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" event={"ID":"69d1f271-0c16-486a-aa84-6ca54c6877e7","Type":"ContainerDied","Data":"9386cf650148a242fc2393f1bb32f50b39e3c0991a2c04affa4a09442fac6d87"} Oct 30 18:45:04.261744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:04.260886 5624 scope.go:122] "RemoveContainer" containerID="d6606522dd52e4666ff56e35d2cb0ad04d11f347bb16f8d84b14fb003c3fbbd3" Oct 30 18:45:04.261744 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:04.261504 5624 scope.go:122] "RemoveContainer" containerID="9386cf650148a242fc2393f1bb32f50b39e3c0991a2c04affa4a09442fac6d87" Oct 30 18:45:04.262167 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:45:04.262125 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2_odh-ai-gateway-infra(69d1f271-0c16-486a-aa84-6ca54c6877e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" Oct 30 18:45:04.863150 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:04.863072 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:05.274900 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:05.274663 5624 scope.go:122] "RemoveContainer" containerID="9386cf650148a242fc2393f1bb32f50b39e3c0991a2c04affa4a09442fac6d87" Oct 30 18:45:05.289078 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:45:05.275231 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2_odh-ai-gateway-infra(69d1f271-0c16-486a-aa84-6ca54c6877e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" Oct 30 18:45:06.430772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:06.430696 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:07.864006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:07.863867 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:09.331329 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:09.331265 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:10.910965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:10.910885 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:12.511359 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:12.511279 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:14.011630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:14.011530 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:14.783771 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:14.783690 5624 scope.go:122] "RemoveContainer" containerID="9386cf650148a242fc2393f1bb32f50b39e3c0991a2c04affa4a09442fac6d87" Oct 30 18:45:15.368446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:15.368249 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" event={"ID":"69d1f271-0c16-486a-aa84-6ca54c6877e7","Type":"ContainerStarted","Data":"c3312715a6e24a5dd195befa31c8711c5b2a72c59c6214804a91d5b1caf6911d"} Oct 30 18:45:15.534165 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:15.534095 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:16.385563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:16.385485 5624 generic.go:356] "Generic (PLEG): container finished" podID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerID="c3312715a6e24a5dd195befa31c8711c5b2a72c59c6214804a91d5b1caf6911d" exitCode=60 Oct 30 18:45:16.385563 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:16.385555 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" event={"ID":"69d1f271-0c16-486a-aa84-6ca54c6877e7","Type":"ContainerDied","Data":"c3312715a6e24a5dd195befa31c8711c5b2a72c59c6214804a91d5b1caf6911d"} Oct 30 18:45:16.386588 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:16.385697 5624 scope.go:122] "RemoveContainer" containerID="9386cf650148a242fc2393f1bb32f50b39e3c0991a2c04affa4a09442fac6d87" Oct 30 18:45:16.386588 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:16.386544 5624 scope.go:122] "RemoveContainer" containerID="c3312715a6e24a5dd195befa31c8711c5b2a72c59c6214804a91d5b1caf6911d" Oct 30 18:45:16.387426 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:45:16.387304 5624 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2_odh-ai-gateway-infra(69d1f271-0c16-486a-aa84-6ca54c6877e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" Oct 30 18:45:16.431030 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:16.430925 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2"] Oct 30 18:45:17.040289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:17.040210 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:17.655929 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:17.655779 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:17.768866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:17.768788 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d1f271-0c16-486a-aa84-6ca54c6877e7-kube-api-access-52j5k\" (UniqueName: \"kubernetes.io/projected/69d1f271-0c16-486a-aa84-6ca54c6877e7-kube-api-access-52j5k\") pod \"69d1f271-0c16-486a-aa84-6ca54c6877e7\" (UID: \"69d1f271-0c16-486a-aa84-6ca54c6877e7\") " Oct 30 18:45:17.769289 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:17.768886 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d1f271-0c16-486a-aa84-6ca54c6877e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d1f271-0c16-486a-aa84-6ca54c6877e7-maas-api-service-ca\") pod \"69d1f271-0c16-486a-aa84-6ca54c6877e7\" (UID: \"69d1f271-0c16-486a-aa84-6ca54c6877e7\") " Oct 30 18:45:17.770265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:17.769652 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d1f271-0c16-486a-aa84-6ca54c6877e7-maas-api-service-ca" pod "69d1f271-0c16-486a-aa84-6ca54c6877e7" (UID: "69d1f271-0c16-486a-aa84-6ca54c6877e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:17.773193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:17.773123 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d1f271-0c16-486a-aa84-6ca54c6877e7-kube-api-access-52j5k" pod "69d1f271-0c16-486a-aa84-6ca54c6877e7" (UID: "69d1f271-0c16-486a-aa84-6ca54c6877e7"). InnerVolumeSpecName "kube-api-access-52j5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:17.871445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:17.871278 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52j5k\" (UniqueName: \"kubernetes.io/projected/69d1f271-0c16-486a-aa84-6ca54c6877e7-kube-api-access-52j5k\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:45:17.871445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:17.871470 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d1f271-0c16-486a-aa84-6ca54c6877e7-maas-api-service-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:45:18.417964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:18.417895 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" Oct 30 18:45:18.418294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:18.417885 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2" event={"ID":"69d1f271-0c16-486a-aa84-6ca54c6877e7","Type":"ContainerDied","Data":"aeaa3ab24c823007446b1bc14bf1c68e1470b8f894511856921615d78adf46e7"} Oct 30 18:45:18.418294 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:18.418172 5624 scope.go:122] "RemoveContainer" containerID="c3312715a6e24a5dd195befa31c8711c5b2a72c59c6214804a91d5b1caf6911d" Oct 30 18:45:18.479867 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:18.479759 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2"] Oct 30 18:45:18.484876 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:18.484818 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-ddcab1-2988976b6kc2"] Oct 30 18:45:18.639641 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:18.639575 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:19.796349 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:19.796155 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" path="/var/lib/kubelet/pods/69d1f271-0c16-486a-aa84-6ca54c6877e7/volumes" Oct 30 18:45:19.938455 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:45:19.938377 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:45:20.310825 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:20.310736 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:21.726874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:21.726800 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:23.227729 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:23.227570 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:24.713304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:24.713236 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:25.536322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.536246 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:25.702460 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.702368 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs"] Oct 30 18:45:25.703291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703241 5624 cpu_manager.go:414] "containerMap: removing container" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.703291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703279 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.703688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703314 5624 cpu_manager.go:414] "containerMap: removing container" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.703688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703325 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.703688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703353 5624 cpu_manager.go:414] "containerMap: removing container" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.703688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703364 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.703688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703632 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.703688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703653 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.703688 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.703668 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d1f271-0c16-486a-aa84-6ca54c6877e7" containerName="cleanup" Oct 30 18:45:25.709865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.709802 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.724914 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.724840 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs"] Oct 30 18:45:25.731676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.731610 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:45:25.731676 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.731657 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:45:25.732149 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.731812 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:25.739363 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.739297 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:45:25.805036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.804821 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.805036 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.804996 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q998\" (UniqueName: \"kubernetes.io/projected/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-kube-api-access-4q998\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.805464 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.805206 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.907019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.906899 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.907438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.907045 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q998\" (UniqueName: \"kubernetes.io/projected/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-kube-api-access-4q998\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.907438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.907179 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.908590 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.908533 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.911672 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.911570 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:25.930120 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:25.930051 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q998\" (UniqueName: \"kubernetes.io/projected/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-kube-api-access-4q998\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:26.038521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:26.038426 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:26.350476 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:26.350268 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:26.391403 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:26.391309 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs"] Oct 30 18:45:26.421736 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:45:26.419180 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f2c7e8_73e1_44ba_82ab_fafd7e33fc7a.slice/crio-ccb3a1662234bd7eebc3e09ba93c9defd600d419298528a221731a19cb9469c1 WatchSource:0}: Error finding container ccb3a1662234bd7eebc3e09ba93c9defd600d419298528a221731a19cb9469c1: Status 404 returned error can't find the container with id ccb3a1662234bd7eebc3e09ba93c9defd600d419298528a221731a19cb9469c1 Oct 30 18:45:26.512837 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:26.512753 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" event={"ID":"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a","Type":"ContainerStarted","Data":"ccb3a1662234bd7eebc3e09ba93c9defd600d419298528a221731a19cb9469c1"} Oct 30 18:45:26.550019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:26.549890 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:45:27.525580 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:27.525504 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" event={"ID":"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a","Type":"ContainerStarted","Data":"79bf0855e65365e5ddd1347eb4c0d0d680a28b793d6aabf82de6f05f7d5bebbf"} Oct 30 18:45:27.569084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:27.568864 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" podStartSLOduration=2.568825116 podStartE2EDuration="2.568825116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:27.566052642 +0000 UTC m=+2621.954580761" watchObservedRunningTime="2026-10-30 18:45:27.568825116 +0000 UTC m=+2621.957353232" Oct 30 18:45:27.924139 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:27.924072 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:28.539626 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:28.539552 5624 generic.go:356] "Generic (PLEG): container finished" podID="90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a" containerID="79bf0855e65365e5ddd1347eb4c0d0d680a28b793d6aabf82de6f05f7d5bebbf" exitCode=0 Oct 30 18:45:28.540456 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:28.539647 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" event={"ID":"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a","Type":"ContainerDied","Data":"79bf0855e65365e5ddd1347eb4c0d0d680a28b793d6aabf82de6f05f7d5bebbf"} Oct 30 18:45:29.513148 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.513081 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:29.851113 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.851031 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:29.874858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.874793 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-xks-ca\") pod \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " Oct 30 18:45:29.875303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.875142 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-openshift-ca\") pod \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " Oct 30 18:45:29.875303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.875218 5624 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-kube-api-access-4q998\" (UniqueName: \"kubernetes.io/projected/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-kube-api-access-4q998\") pod \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\" (UID: \"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a\") " Oct 30 18:45:29.876066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.875660 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-openshift-ca" pod "90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a" (UID: "90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:29.876066 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.875967 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-openshift-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:45:29.878873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.878809 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-kube-api-access-4q998" pod "90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a" (UID: "90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a"). InnerVolumeSpecName "kube-api-access-4q998". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:29.878873 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.878848 5624 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-xks-ca" pod "90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a" (UID: "90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:29.977696 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.977594 5624 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q998\" (UniqueName: \"kubernetes.io/projected/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-kube-api-access-4q998\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:45:29.977696 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:29.977681 5624 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a-maas-api-xks-ca\") on node \"ip-10-0-133-47.ec2.internal\" DevicePath \"\"" Oct 30 18:45:30.539797 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:30.539739 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:30.562531 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:30.562478 5624 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" Oct 30 18:45:30.563029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:30.562464 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs" event={"ID":"90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a","Type":"ContainerDied","Data":"ccb3a1662234bd7eebc3e09ba93c9defd600d419298528a221731a19cb9469c1"} Oct 30 18:45:30.563029 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:30.562701 5624 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb3a1662234bd7eebc3e09ba93c9defd600d419298528a221731a19cb9469c1" Oct 30 18:45:31.116733 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:31.116663 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:32.638377 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:32.638309 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:34.251280 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:34.251223 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:35.705411 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:35.705323 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:37.021156 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:37.021099 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:39.027849 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:39.027781 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:39.130556 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:39.130493 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:39.152019 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:39.151910 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs"] Oct 30 18:45:39.164166 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:39.164083 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ddcab1-qvmbs"] Oct 30 18:45:39.652905 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:39.652847 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:39.794081 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:39.793993 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a" path="/var/lib/kubelet/pods/90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a/volumes" Oct 30 18:45:39.975841 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:39.975636 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:45:44.469160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:44.469097 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:45.278603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:45.278540 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:45.378728 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:45.378652 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:45.623818 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:45.623703 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:49.782106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:49.781926 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:50.007738 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:45:50.007654 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:45:52.392257 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:52.392188 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:45:55.751024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:55.725159 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4bknm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:45:55.751024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:45:55.725273 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4bknm" podUID="50c9434e-7f80-41bf-92c8-c8e8dbb4671b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:46:01.673869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:01.673784 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:01.750397 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:01.750301 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:06.831047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:06.830905 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:07.604414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:07.604310 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:08.350552 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:08.350482 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:09.758541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:09.758432 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:46:13.859423 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:13.859339 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:19.937677 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:46:19.937593 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:46:25.275264 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:25.275174 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:46:32.028964 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:32.028840 5624 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj"] Oct 30 18:46:32.041892 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:32.041792 5624 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec5479-h9nqj"] Oct 30 18:46:33.797766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:33.797659 5624 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e" path="/var/lib/kubelet/pods/dfe88d3f-2ec9-41cf-98e3-5fc7d359c45e/volumes" Oct 30 18:46:34.764865 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:34.764782 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:35.588768 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:35.588694 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:46:40.673874 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:40.673777 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:41.592025 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:41.583063 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:46:44.581493 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:44.581384 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:48.837007 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:48.834134 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:46:50.014898 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:46:50.014790 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:46:55.265049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:55.264884 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:55.364433 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:55.364354 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:56.377546 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:46:56.377472 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:47:04.773871 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:04.773736 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:47:19.932994 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:47:19.932852 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:47:23.078355 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:23.078270 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:47:30.786678 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:30.786596 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:31.266024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.265883 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:31.421973 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.421866 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:31.824222 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.824133 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c"] Oct 30 18:47:31.825132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.825091 5624 cpu_manager.go:414] "containerMap: removing container" podUID="90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a" containerName="revoke-keys" Oct 30 18:47:31.825132 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.825133 5624 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a" containerName="revoke-keys" Oct 30 18:47:31.825346 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.825258 5624 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f2c7e8-73e1-44ba-82ab-fafd7e33fc7a" containerName="revoke-keys" Oct 30 18:47:31.830790 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.830723 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.850745 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.850676 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:47:31.851866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.850676 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:47:31.851866 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.850845 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:47:31.867709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.865301 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:47:31.867709 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.866642 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 18:47:31.871629 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.871535 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c"] Oct 30 18:47:31.886371 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.886291 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-openshift-service-ca\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.886800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.886475 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-maas-api-tls\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.886800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.886715 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fk7d\" (UniqueName: \"kubernetes.io/projected/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-kube-api-access-7fk7d\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.886800 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.886781 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-metrics-certs\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.988730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.988617 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fk7d\" (UniqueName: \"kubernetes.io/projected/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-kube-api-access-7fk7d\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.988730 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.988750 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-metrics-certs\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.989278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.988972 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-openshift-service-ca\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.989278 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.989019 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-maas-api-tls\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.990244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.990200 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-openshift-service-ca\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.993389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.993330 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-maas-api-tls\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:31.993689 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:31.993423 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-metrics-certs\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:32.012404 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:32.012335 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fk7d\" (UniqueName: \"kubernetes.io/projected/bfbb701a-8c6e-46a5-9de0-92438bda3fb1-kube-api-access-7fk7d\") pod \"maas-api-5fc75ddbcb-jxf4c\" (UID: \"bfbb701a-8c6e-46a5-9de0-92438bda3fb1\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:32.158390 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:32.158298 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:32.533603 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:47:32.533524 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbb701a_8c6e_46a5_9de0_92438bda3fb1.slice/crio-7ba7898ad23693a88641b242ed284bc7fe1829fe6d490a34805fc144aeeb3feb WatchSource:0}: Error finding container 7ba7898ad23693a88641b242ed284bc7fe1829fe6d490a34805fc144aeeb3feb: Status 404 returned error can't find the container with id 7ba7898ad23693a88641b242ed284bc7fe1829fe6d490a34805fc144aeeb3feb Oct 30 18:47:32.534716 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:32.534650 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c"] Oct 30 18:47:32.536049 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:32.536005 5624 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:47:33.471324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:33.471245 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" event={"ID":"bfbb701a-8c6e-46a5-9de0-92438bda3fb1","Type":"ContainerStarted","Data":"50ec4bb5fe360feae71b61f1d2b804b4647f11576645eb5177620d5fed463b6f"} Oct 30 18:47:33.472270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:33.471348 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" event={"ID":"bfbb701a-8c6e-46a5-9de0-92438bda3fb1","Type":"ContainerStarted","Data":"7ba7898ad23693a88641b242ed284bc7fe1829fe6d490a34805fc144aeeb3feb"} Oct 30 18:47:33.472270 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:33.471565 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:33.517856 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:33.517735 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" podStartSLOduration=2.046907892 podStartE2EDuration="2.517701335s" totalImagesPullingTime="470.793443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:33.515396162 +0000 UTC m=+2747.903924288" watchObservedRunningTime="2026-10-30 18:47:33.517701335 +0000 UTC m=+2747.906229451" Oct 30 18:47:34.391214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:34.391139 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:34.496628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:34.496554 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:37.433350 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:37.433267 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:37.538854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:37.538795 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:37.633725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:37.633664 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:39.116426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:39.116303 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:39.490777 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:39.490719 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-jxf4c" Oct 30 18:47:40.628793 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:40.628712 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:40.977727 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:40.977505 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:47:42.255292 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:42.255224 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:43.945103 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:43.945022 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:44.707503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:44.707419 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:45.555160 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:45.555099 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:47.266015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:47.265845 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:48.829561 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:48.829455 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:49.937191 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:47:49.937112 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:47:50.429724 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:50.429661 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:51.959244 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:51.959151 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:53.557362 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:53.557283 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:55.143261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:55.143158 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:55.956861 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:55.956766 5624 scope.go:122] "RemoveContainer" containerID="0979e4c7e9731b8b6e7c59f4f839bff44822e528f67292467310f36dad7b3e47" Oct 30 18:47:55.974239 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:55.974196 5624 scope.go:122] "RemoveContainer" containerID="4a1258dea0b710410272190eb82e388341f175a353ba7dfcc19d62507912def2" Oct 30 18:47:56.816185 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:56.816108 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:58.356434 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:58.356358 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:59.848516 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:47:59.848461 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:00.884285 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:00.884210 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:00.982600 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:00.982535 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:01.333129 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:01.332963 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:02.693831 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:02.693769 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:03.147634 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:03.147553 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:03.249917 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:03.249829 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:05.387422 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:05.387345 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:48:05.506720 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:05.506636 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:06.529898 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:06.529802 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:07.806539 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:07.806464 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:08.303620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:08.303542 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:08.400833 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:08.400739 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:12.300258 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:12.300185 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:12.404301 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:12.404235 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:13.792549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:13.792459 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:13.882048 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:13.881858 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:13.990261 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:13.990176 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:14.084181 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:14.083983 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:14.186882 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:14.186785 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:15.473257 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:15.473182 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:16.322761 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:16.322676 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:48:17.240498 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:17.240418 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:18.798046 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:18.797876 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:19.938541 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:48:19.938454 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:48:20.491991 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:20.491846 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:22.345300 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:22.345254 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:24.044669 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:24.044477 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:25.640446 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:25.640346 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:27.244823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:27.244747 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:27.385179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:27.385096 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:29.092109 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:29.092019 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:31.044447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:31.044248 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:32.587299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:32.587241 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:34.267301 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:34.267246 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:35.868965 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:35.868865 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:37.481868 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:37.481770 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:39.232008 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:39.231892 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:40.776445 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:40.776305 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:42.476836 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:42.476740 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:44.164606 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:44.164538 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:44.846229 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:44.846156 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:45.824503 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:45.824412 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:47.381619 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:47.381556 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:48.976452 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:48.976384 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:49.938597 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:48:49.938515 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:48:50.678024 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:50.677903 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:52.254549 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:52.254470 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:53.960006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:53.959898 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:55.648414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:55.648318 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:57.262538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:57.262468 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:58.834581 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:48:58.834513 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:00.468304 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:00.468240 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:02.308489 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:02.308408 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:04.131161 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:04.131040 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:05.769982 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:05.769862 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:07.369255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:07.369164 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:09.101100 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:09.101004 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:10.206150 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:10.206079 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:10.764221 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:10.764142 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:12.366177 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:12.366110 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:13.950328 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:13.950270 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:15.666106 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:15.666029 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.390521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:17.390413 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:18.878897 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:18.878829 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:19.934708 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:49:19.934626 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:49:20.471536 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:20.471465 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:22.165360 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:22.165282 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.828911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:23.820456 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:25.349630 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:25.349556 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:27.068704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:27.068616 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:28.907712 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:28.907631 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:30.736000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:30.735903 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:32.281187 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:32.281113 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:33.973541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:33.973469 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:35.644541 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:35.644377 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:37.253108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:37.252924 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:38.988654 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:38.988561 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:40.759367 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:40.759276 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:42.273615 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:42.273532 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:43.860059 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:43.859980 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:44.943766 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:44.943697 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:45.473838 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:45.473767 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:47.240084 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:47.239622 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:49.180910 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:49.180847 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:49.937712 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:49:49.937639 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:49:50.940734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:50.939232 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:52.577907 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:52.577818 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:54.358684 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:54.358519 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:55.965774 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:55.965704 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:57.717627 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:57.717566 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:58.268789 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:58.268697 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:58.500589 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:58.500501 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:59.539142 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:49:59.538967 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:05.951725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:05.951632 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:06.419562 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:06.419462 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:06.543909 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:06.543815 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:07.128667 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:07.128581 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:09.280848 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:09.280747 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:50:13.097212 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:13.097152 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:13.204823 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:13.204754 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:13.438714 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:13.438642 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:16.845409 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:16.845314 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:50:20.177464 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:50:20.177348 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:50:20.206015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:20.198207 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:20.206015 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:20.198342 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:20.206834 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:20.206742 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:20.214067 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:20.213404 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:24.684538 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:24.684463 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:50:30.165776 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:30.165678 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:30.317047 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:30.316916 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:42.875040 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:42.874922 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:50:45.066206 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:45.066140 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:45.766500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:45.766427 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:50.074334 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:50:50.074276 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:50:50.207985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:50.207605 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h2w59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:50.207985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:50.207772 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h2w59" podUID="357250e3-7a1d-404b-be85-694272143fea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:50.211990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:50.210278 5624 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-f5zxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:50:50.211990 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:50.210417 5624 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-f5zxr" podUID="f46323fd-bfd5-43e1-82c5-fd2b4766e38e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:50:58.083313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:50:58.083253 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:05.080755 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:05.080687 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:09.367313 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:09.367258 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cb9789cff-xpn9q_81799468-0dff-48d9-ba4c-dc4d0c29cf40/authorino/0.log" Oct 30 18:51:09.476521 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:09.476426 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2hj6q"] Oct 30 18:51:19.939667 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:51:19.939589 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:51:19.996818 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:19.996731 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bcd7466d5-kzdwm_4326dbaf-621c-4d40-9f86-e29dbdd1b649/manager/0.log" Oct 30 18:51:21.241333 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:21.241281 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-49l6m_0a09133a-00c6-4632-ba5e-b859f8da282e/server/0.log" Oct 30 18:51:21.446418 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:21.446348 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-84jnk_dc1ecec4-614d-4cbf-978b-7b0fdb293d0f/manager/0.log" Oct 30 18:51:22.419017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:22.418929 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fc75ddbcb-jxf4c_bfbb701a-8c6e-46a5-9de0-92438bda3fb1/maas-api/0.log" Oct 30 18:51:25.436494 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:25.436347 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cb9789cff-xpn9q_81799468-0dff-48d9-ba4c-dc4d0c29cf40/authorino/0.log" Oct 30 18:51:25.694665 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:25.694486 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r8j74_54747d07-9037-4c21-be5c-c554f0b21b5d/manager/0.log" Oct 30 18:51:25.894879 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:25.894800 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kl2vt_75ee8c7b-b877-4f5c-9b74-08bda044c20c/manager/0.log" Oct 30 18:51:26.298000 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:26.297903 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7295w_a9933a5a-2dec-4927-9bb8-c38905943483/manager/0.log" Oct 30 18:51:26.511193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:26.511108 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2hj6q_5c4422bc-c3e1-48f1-99a8-c62260e1d25c/limitador/0.log" Oct 30 18:51:26.700376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:26.700297 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mndc5_09728775-6be0-45a5-bfb9-480ff7a90759/manager/1.log" Oct 30 18:51:27.138273 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:27.138192 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mndc5_09728775-6be0-45a5-bfb9-480ff7a90759/manager/0.log" Oct 30 18:51:27.700317 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:27.700247 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-xvbrd_a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7/istio-proxy/0.log" Oct 30 18:51:27.912078 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:27.911911 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4drwq_9249b0ef-7632-4d51-9254-5e0e95c82313/discovery/0.log" Oct 30 18:51:28.490389 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:28.490323 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-4bknm_50c9434e-7f80-41bf-92c8-c8e8dbb4671b/istio-proxy/0.log" Oct 30 18:51:28.686115 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:28.686046 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-f5zxr_f46323fd-bfd5-43e1-82c5-fd2b4766e38e/istio-proxy/0.log" Oct 30 18:51:28.896734 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:28.896683 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-h2w59_357250e3-7a1d-404b-be85-694272143fea/istio-proxy/0.log" Oct 30 18:51:30.380660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:30.380602 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn_6a1fa476-56bf-4fd4-a676-5da6109ab2ed/istio-proxy/0.log" Oct 30 18:51:30.741739 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:30.741552 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:51:31.777804 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:31.777707 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz_79e88ba7-0418-4268-b1b3-e50946e634eb/main/0.log" Oct 30 18:51:31.974324 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:31.974238 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7gjfw_e069f9a5-f97d-45a9-a64a-d47a4118f251/main/0.log" Oct 30 18:51:32.368901 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:32.368829 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-lz5rk_eba05bdf-704e-4dc8-9da9-3e52b6f0a285/main/0.log" Oct 30 18:51:32.744987 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:32.744867 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-7wj9m_d2333eaa-171b-4dd4-a978-24ba23defed7/main/0.log" Oct 30 18:51:32.938488 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:32.938420 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk_32a32196-f4a2-41c3-b028-25e792821d05/main/0.log" Oct 30 18:51:33.123158 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:33.123028 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd_da2b19fd-2648-4e80-bde1-16211807bc16/main/0.log" Oct 30 18:51:34.016673 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:34.016597 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cb9789cff-xpn9q_81799468-0dff-48d9-ba4c-dc4d0c29cf40/authorino/0.log" Oct 30 18:51:43.236329 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:43.236262 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5bcd7466d5-kzdwm_4326dbaf-621c-4d40-9f86-e29dbdd1b649/manager/0.log" Oct 30 18:51:44.313880 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:44.313809 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-49l6m_0a09133a-00c6-4632-ba5e-b859f8da282e/server/0.log" Oct 30 18:51:44.511772 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:44.511702 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-84jnk_dc1ecec4-614d-4cbf-978b-7b0fdb293d0f/manager/0.log" Oct 30 18:51:45.171537 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:45.171458 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:45.275255 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:45.275205 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fc75ddbcb-jxf4c_bfbb701a-8c6e-46a5-9de0-92438bda3fb1/maas-api/0.log" Oct 30 18:51:47.756704 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:47.756633 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cb9789cff-xpn9q_81799468-0dff-48d9-ba4c-dc4d0c29cf40/authorino/0.log" Oct 30 18:51:48.001450 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:48.001379 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r8j74_54747d07-9037-4c21-be5c-c554f0b21b5d/manager/0.log" Oct 30 18:51:48.202646 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:48.202587 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kl2vt_75ee8c7b-b877-4f5c-9b74-08bda044c20c/manager/0.log" Oct 30 18:51:48.597651 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:48.597436 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7295w_a9933a5a-2dec-4927-9bb8-c38905943483/manager/0.log" Oct 30 18:51:48.787395 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:48.787341 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2hj6q_5c4422bc-c3e1-48f1-99a8-c62260e1d25c/limitador/0.log" Oct 30 18:51:48.983291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:48.983197 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mndc5_09728775-6be0-45a5-bfb9-480ff7a90759/manager/1.log" Oct 30 18:51:49.543006 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:49.542927 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-xvbrd_a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7/istio-proxy/0.log" Oct 30 18:51:49.741955 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:49.741863 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4drwq_9249b0ef-7632-4d51-9254-5e0e95c82313/discovery/0.log" Oct 30 18:51:49.935333 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:51:49.935278 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:51:50.313579 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:50.313395 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-4bknm_50c9434e-7f80-41bf-92c8-c8e8dbb4671b/istio-proxy/0.log" Oct 30 18:51:50.504564 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:50.504492 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-f5zxr_f46323fd-bfd5-43e1-82c5-fd2b4766e38e/istio-proxy/0.log" Oct 30 18:51:50.701322 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:50.701267 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-h2w59_357250e3-7a1d-404b-be85-694272143fea/istio-proxy/0.log" Oct 30 18:51:52.276695 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:52.276618 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn_6a1fa476-56bf-4fd4-a676-5da6109ab2ed/istio-proxy/0.log" Oct 30 18:51:52.655458 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:52.655402 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:51:53.403052 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:53.402984 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9nrcz_79e88ba7-0418-4268-b1b3-e50946e634eb/main/0.log" Oct 30 18:51:53.601603 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:53.601543 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7gjfw_e069f9a5-f97d-45a9-a64a-d47a4118f251/main/0.log" Oct 30 18:51:53.973399 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:53.973323 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-lz5rk_eba05bdf-704e-4dc8-9da9-3e52b6f0a285/main/0.log" Oct 30 18:51:54.371752 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:54.371576 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-7wj9m_d2333eaa-171b-4dd4-a978-24ba23defed7/main/0.log" Oct 30 18:51:54.551542 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:54.551473 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-vz5dk_32a32196-f4a2-41c3-b028-25e792821d05/main/0.log" Oct 30 18:51:54.736038 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:54.735854 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xbqdd_da2b19fd-2648-4e80-bde1-16211807bc16/main/0.log" Oct 30 18:51:56.074829 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:51:56.074747 5624 scope.go:122] "RemoveContainer" containerID="79bf0855e65365e5ddd1347eb4c0d0d680a28b793d6aabf82de6f05f7d5bebbf" Oct 30 18:52:19.936074 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:52:19.935994 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:52:30.942820 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:52:30.942758 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:52:31.134902 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:52:31.134851 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:52:39.798299 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:52:39.798242 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:45.298891 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:52:45.298802 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:49.932703 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:52:49.932627 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:53:17.824431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:17.824359 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:53:18.014764 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:18.014710 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:53:18.361632 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:18.361547 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:18.470107 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:18.470031 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:19.933327 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:53:19.933262 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:53:23.011291 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:23.011229 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qg2lp_8e9cf883-7d59-46f0-b457-f84c7c678251/global-pull-secret-syncer/0.log" Oct 30 18:53:23.603414 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:23.603352 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" Oct 30 18:53:24.058444 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:24.058377 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-47.ec2.internal_05806b328c870eba3d65f2b11ca3bf01/haproxy/0.log" Oct 30 18:53:30.369565 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:30.369492 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cb9789cff-xpn9q_81799468-0dff-48d9-ba4c-dc4d0c29cf40/authorino/0.log" Oct 30 18:53:30.461063 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:30.460979 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-r8j74_54747d07-9037-4c21-be5c-c554f0b21b5d/manager/0.log" Oct 30 18:53:30.564093 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:30.564036 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kl2vt_75ee8c7b-b877-4f5c-9b74-08bda044c20c/manager/0.log" Oct 30 18:53:30.810911 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:30.810835 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7295w_a9933a5a-2dec-4927-9bb8-c38905943483/manager/0.log" Oct 30 18:53:30.890995 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:30.890915 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2hj6q_5c4422bc-c3e1-48f1-99a8-c62260e1d25c/limitador/0.log" Oct 30 18:53:31.028370 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:31.028294 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mndc5_09728775-6be0-45a5-bfb9-480ff7a90759/manager/0.log" Oct 30 18:53:31.051854 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:31.051781 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-mndc5_09728775-6be0-45a5-bfb9-480ff7a90759/manager/1.log" Oct 30 18:53:32.738286 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.738215 5624 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr"] Oct 30 18:53:32.744226 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.744163 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.765340 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.765286 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jtmf9\"/\"openshift-service-ca.crt\"" Oct 30 18:53:32.765620 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.765363 5624 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jtmf9\"/\"kube-root-ca.crt\"" Oct 30 18:53:32.779233 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.779168 5624 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jtmf9\"/\"default-dockercfg-6tdcw\"" Oct 30 18:53:32.786601 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.786502 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr"] Oct 30 18:53:32.814196 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.814103 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/98574237-4f76-4365-aced-eed783c80aca-kube-api-access-msg64\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.814578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.814369 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-proc\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.814578 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.814498 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-lib-modules\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.814815 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.814657 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-podres\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.814858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.814822 5624 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-sys\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.916628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.916541 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-sys\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.916628 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.916657 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/98574237-4f76-4365-aced-eed783c80aca-kube-api-access-msg64\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.917252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.916753 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-sys\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.917252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.916756 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-proc\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.917252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.916826 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-proc\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.917252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.916883 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-lib-modules\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.917252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.917038 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-lib-modules\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.917252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.917049 5624 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-podres\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.917252 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.917188 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98574237-4f76-4365-aced-eed783c80aca-podres\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:32.940809 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:32.940736 5624 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msg64\" (UniqueName: \"kubernetes.io/projected/98574237-4f76-4365-aced-eed783c80aca-kube-api-access-msg64\") pod \"perf-node-gather-daemonset-htgpr\" (UID: \"98574237-4f76-4365-aced-eed783c80aca\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:33.068376 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:33.068175 5624 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:33.242419 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:33.242345 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-brcll_e2a6cc3b-391d-4626-908a-5466994e21d5/kube-state-metrics/0.log" Oct 30 18:53:33.346214 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:33.346021 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-brcll_e2a6cc3b-391d-4626-908a-5466994e21d5/kube-rbac-proxy-main/0.log" Oct 30 18:53:33.430447 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:33.430373 5624 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr"] Oct 30 18:53:33.432676 ip-10-0-133-47 kubenswrapper[5624]: W1030 18:53:33.432608 5624 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98574237_4f76_4365_aced_eed783c80aca.slice/crio-5655080e7c6df53d926eaef5f64bf1cbafe349c028df622880ce84633c0afe27 WatchSource:0}: Error finding container 5655080e7c6df53d926eaef5f64bf1cbafe349c028df622880ce84633c0afe27: Status 404 returned error can't find the container with id 5655080e7c6df53d926eaef5f64bf1cbafe349c028df622880ce84633c0afe27 Oct 30 18:53:33.469327 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:33.469278 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-brcll_e2a6cc3b-391d-4626-908a-5466994e21d5/kube-rbac-proxy-self/0.log" Oct 30 18:53:33.692265 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:33.692211 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-f87jc_9e62ca62-c183-4859-8d52-a52aa8182c73/monitoring-plugin/0.log" Oct 30 18:53:34.130969 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.130905 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bjq6_3275088d-42df-41bd-8b42-b739a353962d/node-exporter/0.log" Oct 30 18:53:34.189398 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.189309 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" event={"ID":"98574237-4f76-4365-aced-eed783c80aca","Type":"ContainerStarted","Data":"bb793eead705aa511235e735d6478009525aa48aa30c8965620c419b9e2d9510"} Oct 30 18:53:34.189810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.189445 5624 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" event={"ID":"98574237-4f76-4365-aced-eed783c80aca","Type":"ContainerStarted","Data":"5655080e7c6df53d926eaef5f64bf1cbafe349c028df622880ce84633c0afe27"} Oct 30 18:53:34.189810 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.189488 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:34.234844 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.234782 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bjq6_3275088d-42df-41bd-8b42-b739a353962d/kube-rbac-proxy/0.log" Oct 30 18:53:34.237190 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.237060 5624 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" podStartSLOduration=2.237016692 podStartE2EDuration="2.237016692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:34.232719693 +0000 UTC m=+3108.621247811" watchObservedRunningTime="2026-10-30 18:53:34.237016692 +0000 UTC m=+3108.625544809" Oct 30 18:53:34.338660 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.338609 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7bjq6_3275088d-42df-41bd-8b42-b739a353962d/init-textfile/0.log" Oct 30 18:53:34.762427 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.762374 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pjv29_bf49c75c-fb11-4c2d-ac07-b41d85f8f106/kube-rbac-proxy-main/0.log" Oct 30 18:53:34.866786 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.866729 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pjv29_bf49c75c-fb11-4c2d-ac07-b41d85f8f106/kube-rbac-proxy-self/0.log" Oct 30 18:53:34.969302 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:34.969254 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pjv29_bf49c75c-fb11-4c2d-ac07-b41d85f8f106/openshift-state-metrics/0.log" Oct 30 18:53:35.086124 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:35.085882 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ade95085-ca7b-4e3b-9883-9dca382e1d69/prometheus/0.log" Oct 30 18:53:35.189179 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:35.189107 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ade95085-ca7b-4e3b-9883-9dca382e1d69/config-reloader/0.log" Oct 30 18:53:35.292056 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:35.291998 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ade95085-ca7b-4e3b-9883-9dca382e1d69/thanos-sidecar/0.log" Oct 30 18:53:35.396869 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:35.396803 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ade95085-ca7b-4e3b-9883-9dca382e1d69/kube-rbac-proxy-web/0.log" Oct 30 18:53:35.499500 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:35.499452 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ade95085-ca7b-4e3b-9883-9dca382e1d69/kube-rbac-proxy/0.log" Oct 30 18:53:35.619957 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:35.619902 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ade95085-ca7b-4e3b-9883-9dca382e1d69/kube-rbac-proxy-thanos/0.log" Oct 30 18:53:35.733858 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:35.733703 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ade95085-ca7b-4e3b-9883-9dca382e1d69/init-config-reloader/0.log" Oct 30 18:53:36.489906 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:36.489840 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f988c6b9-jw4rr_269d1132-4f1d-4b95-b545-2b2659d245a7/thanos-query/0.log" Oct 30 18:53:36.592310 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:36.592270 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f988c6b9-jw4rr_269d1132-4f1d-4b95-b545-2b2659d245a7/kube-rbac-proxy-web/0.log" Oct 30 18:53:36.694725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:36.694671 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f988c6b9-jw4rr_269d1132-4f1d-4b95-b545-2b2659d245a7/kube-rbac-proxy/0.log" Oct 30 18:53:36.801017 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:36.800834 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f988c6b9-jw4rr_269d1132-4f1d-4b95-b545-2b2659d245a7/prom-label-proxy/0.log" Oct 30 18:53:36.904303 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:36.904259 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f988c6b9-jw4rr_269d1132-4f1d-4b95-b545-2b2659d245a7/kube-rbac-proxy-rules/0.log" Oct 30 18:53:37.009213 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:37.009113 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f988c6b9-jw4rr_269d1132-4f1d-4b95-b545-2b2659d245a7/kube-rbac-proxy-metrics/0.log" Oct 30 18:53:40.217426 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:40.217359 5624 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-htgpr" Oct 30 18:53:43.728276 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:43.728205 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6mcgq_b1cd5923-8633-4b24-8310-e06ac0a2de68/dns/0.log" Oct 30 18:53:43.867643 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:43.867586 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6mcgq_b1cd5923-8633-4b24-8310-e06ac0a2de68/kube-rbac-proxy/0.log" Oct 30 18:53:44.589117 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:44.589035 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n97j8_3b8bc620-2f0c-4060-b76e-2b2365e3621f/dns-node-resolver/0.log" Oct 30 18:53:45.397424 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:45.397355 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-xvbrd_a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7/istio-proxy/0.log" Oct 30 18:53:45.970985 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:45.970875 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l6vjs_c3b4bb1c-34fd-44ba-8f19-1df9ab9f2aa2/node-ca/0.log" Oct 30 18:53:46.386168 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:46.386097 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:46.796193 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:46.796000 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4drwq_9249b0ef-7632-4d51-9254-5e0e95c82313/discovery/0.log" Oct 30 18:53:46.982624 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:46.982552 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-4bknm_50c9434e-7f80-41bf-92c8-c8e8dbb4671b/istio-proxy/0.log" Oct 30 18:53:47.044683 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:47.044611 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-f5zxr_f46323fd-bfd5-43e1-82c5-fd2b4766e38e/istio-proxy/0.log" Oct 30 18:53:47.132085 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:47.132028 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-h2w59_357250e3-7a1d-404b-be85-694272143fea/istio-proxy/0.log" Oct 30 18:53:47.695387 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:47.695313 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn_6a1fa476-56bf-4fd4-a676-5da6109ab2ed/istio-proxy/0.log" Oct 30 18:53:47.829137 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:47.829087 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:53:47.864321 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:47.864273 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-xvbrd_a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7/istio-proxy/0.log" Oct 30 18:53:49.278830 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:49.278748 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4drwq_9249b0ef-7632-4d51-9254-5e0e95c82313/discovery/0.log" Oct 30 18:53:49.619743 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:49.619585 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-4bknm_50c9434e-7f80-41bf-92c8-c8e8dbb4671b/istio-proxy/0.log" Oct 30 18:53:49.731607 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:49.731516 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-f5zxr_f46323fd-bfd5-43e1-82c5-fd2b4766e38e/istio-proxy/0.log" Oct 30 18:53:49.844438 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:49.844371 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-h2w59_357250e3-7a1d-404b-be85-694272143fea/istio-proxy/0.log" Oct 30 18:53:49.934700 ip-10-0-133-47 kubenswrapper[5624]: E1030 18:53:49.934636 5624 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x6tdb" containerName="extractor" Oct 30 18:53:50.715922 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:50.715843 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn_6a1fa476-56bf-4fd4-a676-5da6109ab2ed/istio-proxy/0.log" Oct 30 18:53:50.948301 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:50.948258 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:53:51.164301 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:51.164247 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-xvbrd_a246d0ff-5e96-46e0-9e23-e3ebdf7b95d7/istio-proxy/0.log" Oct 30 18:53:52.537502 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:52.537435 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4drwq_9249b0ef-7632-4d51-9254-5e0e95c82313/discovery/0.log" Oct 30 18:53:52.722808 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:52.722748 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-4bknm_50c9434e-7f80-41bf-92c8-c8e8dbb4671b/istio-proxy/0.log" Oct 30 18:53:52.791108 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:52.790891 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-f5zxr_f46323fd-bfd5-43e1-82c5-fd2b4766e38e/istio-proxy/0.log" Oct 30 18:53:52.857896 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:52.857797 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-h2w59_357250e3-7a1d-404b-be85-694272143fea/istio-proxy/0.log" Oct 30 18:53:53.216262 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:53.216210 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9l55k_630626e4-d3eb-4e14-8c9d-9363718e5049/serve-healthcheck-canary/0.log" Oct 30 18:53:53.380725 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:53.380656 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-q6tbn_6a1fa476-56bf-4fd4-a676-5da6109ab2ed/istio-proxy/0.log" Oct 30 18:53:53.508474 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:53.508323 5624 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-j6n87_d13122e5-4950-40fc-8bb5-046b9fa77766/payload-processing/0.log" Oct 30 18:53:53.678431 ip-10-0-133-47 kubenswrapper[5624]: I1030 18:53:53.678342 5624 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6w92c_5d05019b-efc6-444a-986d-6f63420ba3ff/konnectivity-agent/0.log" limit=1024